Eye-Perspective View Management for Optical See-Through Head-Mounted Displays

2.1 View Management

A number of approaches have been proposed that use the real-world background to improve AR rendering. Relevant to this work are those that improve the annotations’ layout, i.e., view management, and rendering to improve legibility.

Annotation Placement.. In early work on view management, Bell et al. [4] optimized label placement in AR and overcame the issue of EPR by simulating their approach in VR. Later work utilized VST AR, where the user perceives the real world via a video feed that can be analyzed and augmented directly as the user’s view. Examples include the optimization of annotation placement based on visual features or visual saliency [17, 44], snapping to or aligning with geometric features  [26, 39], or the adaptation of label visibility based on a background analysis [17, 28, 40]. For instance, Orlosky et al. [40] utilizes background uniformity and darkness criteria to identify areas where labels would be legible in a video image. However, these approaches have been demonstrated in VST AR as applying these methods directly to OST HMDs produces erroneous results, because the camera view(s) and, thus, the base for optimization, do not match the user’s view of the scene (Fig. 3).

Contrast and Color Enhancement.. Particularly in the area of contrast and color enhancement, several approaches have considered the effect of blending displayed information with the real-world background [22, 25, 27, 49]. However, most of these works rely on simulations to demonstrate their approach and do not solve the issue of mapping from a camera view to the OST display as seen by the users. An exception to this is the work of Langlotz et al. [25, 27] who modify an OST HMD with semitransparent mirrors reflecting incoming light towards calibrated cameras to enable EPR in order to compensate for the effect of background color blending by applying pixel-wise corrections. While they proved the efficacy of the approach, it requires hardware modifications that substantially add to the size of the HMD.

Stereo view management. There are several examples in the literature on utilizing depth cues and stereo perception for label placement (e.g. [42, 43]). Recently, there has also been work on identifying and overcoming depth issues and perceptual conflicts between text and background when displaying subtitles in VR scenes [38, 48]. When labels are not aligned with scene geometry but shown at a different depth, double vision of the background negatively impacts legibility. Furthermore, stereo view management will perform worse in VST AR HMDs because the background negatively affects the legibility of labels, with authors reporting a decrease in performance in search tasks [42]. Overcoming this issue requires understanding the scene as viewed by the user which seemed so far unfeasible for OST HMD.

2.2 Eye- and User-Perspective Rendering in AR

UPR or EPR for AR is typically aimed at VST AR displays and only very few considered related issues for OST HMDs.

Video See-Through AR Displays.. Video-based UPR approaches attempt to warp or render a view of a physical scene so that it appears as seen by the user of a system, for instance, based on calculating the desired viewport for a user’s eye position when looking through a handheld AR display  [21, 31]. Samini et al. [45, 46] demonstrated the feasibility of this UPR technique for interaction and search tasks. A variation of this approach was presented by Tomioka et al. [54] who used 3D feature points of the scene to continuously calculate a homography that transforms the video image to the desired view of the user. To avoid continuous updates and the performance overhead of head tracking, Pucihar et al. [56, 57] assumed a fixed location of the user’s head relative to the handheld display. Mohr et al. [35] overcame the restricted head position by efficiently updating the head position only when the user’s viewpoint relative to the handheld device changed beyond a certain threshold. 2D approaches that warp the camera feed introduce perceivable artifacts if the scene is not mostly planar, or introduce disocclusion artifacts where parts of the scene are not visible from the user’s viewpoint. This can be overcome by using image data and proxy geometry from RGBD cameras [47] or a complete 3D reconstruction [3]. Due to the computational overhead of these methods, other works rely on a single depth image and create missing information by image-based rendering (IBR) [1, 2, 47].

Closest to our technical solutions for realizing EPR is the work of Chaurasia et al. [7] who presented a UPR method to create a video see-through AR mode on an Oculus Quest, a mobile VR platform, rendering the information of greyscale stereo cameras from the view of the user’s eyes. A very recent work aimed at providing an improved 3D reconstruction. Instead of using motion vectors for creating the stereo matching [7], the approach used learning-based stereo matching [29]. Unfortunately, two high-end graphics cards (NVidia Titan) are needed for computing the see-through mode, which makes it unlikely that the method will work for mobile AR hardware soon. Finally, the existing approaches have not been applied in OST HMDs or compared against each other in practical scenarios. In this paper, we perform a systematic evaluation of EPR approaches that are based on UPR to determine feasible solutions for synthesizing the user’s view through OST HMDs to perform optimizations based on the real-world background.

Figure 4:

View Figure

Optical See-Through AR Displays.. To prototype and evaluate methods that rely on the user’s viewpoint, previous work has often relied on software simulations of OST displays  [22], or cameras looking through the display instead of the user’s eyes [23]. A solution that allows for human subject experiments, is the approach of Langlotz et al. [25, 27, 50] which captures the user’s view by introducing additional cameras and beam splitter into a prototypical HMD. While demonstrated in a mobile setup [50] their approach adds to the bulk and weight of the HMD, in particular when supporting larger fields of view. We base our EPR on the idea of UPR for mobile devices such as smartphones or tablets.

4.1 Eye-Perspective Rendering

While EPR has not been proposed for OST HMDs, we benefit from the works presenting UPR in the context of handheld VST AR displays. Hence, in the first step, we implement several EPR approaches that are inspired by the current state-of-the-art technology in UPR. To increase the practical applicability of EPR to real-world use cases, we focused our exploration on algorithms that can be implemented on consumer-grade hardware without additional hardware modifications or excessive hardware requirements. Therefore, approaches that require extensive modifications of the HMD [25] or need a dedicated top-end graphic card just for computing the EPR for one eye [29] were not considered.

To facilitate the discussion of EPR, in the following, we define the general EPR system (Fig. 6). An EPR system consists of an HMD worn by a user and positioned in the world coordinate system W. Rigidly attached to the HMD is an RGBD sensor that is also used for tracking. T_H is used to transform from world coordinates to the coordinate system of the RGBD sensor and, thus, the head coordinate system H. Our system synthesizes views of the real-world scene for the user’s eyes. Hence, the scene must be transformed to the eye coordinate system E_i, where i can be left or right eye. We achieve this by using the transformation \(T_{E_i}\) for each eye to move from H to E_i. For view synthesis, we require input images I_H for EPR captured using the RGBD sensor. For our discussion, we distinguish between the color component of this image I_{H, RGB} and the depth component of the image I_{H, D}. The synthesized images for E_i are defined as \(I_{E_i}\).

Based on the UPR literature, we decided to utilize two general approaches to synthesize the user’s view \(I_{E_i}\) through the display: a homography of the currently captured view I_{H, RGB}, and using 3D proxy geometry and visual information of one [47] or more [2] captured views I_{H, RGBD}. We implemented three EPR approaches (Fig. 6): (1) a homography of the video I_{H, RGB} based on an analysis of the scene that identifies the dominant plane, (2) view synthesis using proxy geometry based on the depth and reprojection of I_{H, RGBD} into the user’s view, and (3) view synthesis using the same proxy and IBR for view synthesis. As the semi-transparent display material of an OST HMD modifies the scene colors passing through the display [22], the final EPR of each method is modified by a constant factor that depends on the used HMDs. In the following, we discuss details of these approaches.

Homography.. An EPR view synthesis can be achieved utilizing 3D information of sparse SLAM-based reconstructions and creating a homography of the captured image I_{H, RGB}  [54]. We create a homography of I_{H, RGB} based on 3D points of the largest identified plane in the scene. To ensure temporal coherence, the homography input preferably considers a stable set of points that was visible over multiple frames and, thus has high confidence. An advantage of the homography is that it does not necessarily require a depth camera as SLAM features are sufficient for calculating the homography. A homography method works well in scenes containing mainly planar geometry at close distances, or with more complex scenes when viewed from a distance [5]. As the homography uses the current live video I_{H, RGB}, the method can represent changes to the scene structure when planarity or distance constraints are intact.

Reprojection.. The reprojection method utilizes depth and color information of the input image I_{H, RGBD} to synthesize the user’s view of the real world. The live RGB image is projected onto the depth map and the textured geometry is transformed by T_E into the user’s view. The reprojection method is suitable for structured 3D scenes but may suffer from disocclusion artifacts leading to missing depth and color information in the synthesized view I_E. As the method utilizes currently available scene information using I_RGBD, it can react to changes in the scene during the interaction. However, view-dependent visual changes cannot be modeled.

Image-based Rendering.. The IBR method also transforms the available 3D proxy geometry into the user’s view. As depth cameras may have issues with certain materials (e.g., Azure DK), or homogeneously textured areas (e.g., Stereo ZED Mini), a depth inpainting method is used to fill in small holes in the depth map before reprojection, and again after reprojection to fill in holes due to disocclusion artifacts. Our inpainting method is inspired by Schöps et al. [47] but uses an efficient push-pull approach for inpainting [16]. An IBR algorithm [6] fills in missing color information using images that are recorded while the user interacts with the scene. The IBR method can be applied to complex 3D scenes and allows us to fill in missing information for the user’s view with recorded scene information. View-dependent visual changes can also be taken into account when synthesizing views. However, when scene changes occur, the strategy recording I_{H, RGB} views for the IBR must ensure that images are kept up to date.

Figure 7:

View Figure

4.2 View Management

Once we have computed the individual view for each eye using EPR we need to optimize the annotations placements in these views. To do so, we designed a view management method that adapts the layout based on the reliability of EPR information. Our view management is based on state-of-the-art hedgehog labeling [30, 52] that allows us to easily define criteria for view management and provides a temporally coherent layout.

Principle. Labels represent 3D objects and are placed in 3D world space. The layout algorithm projects 3D labels into the user’s eyes using the determined eye position to determine a 2D projection of the label. This 2D projection is used to access the information calculated based on the EPR view for a user’s eye in order to optimize the label position. The label position is adjusted using a vector field calculated from a set of optimization criteria based on the EPR views, adapting the layout based on information from both of the user’s eyes. After the optimization is finished for the current view, the calculated positions are converted back to 3D space via unprojection. In line with Tatzgern et al. [52], the layout algorithm places labels close to their origin, and the label movement is constrained to 3D planes in the scene. Planes are placed at the depth of the label origin which typically corresponds to the 3D position of the annotated object. To ensure temporal coherence of labels, as suggested by Tatzgern et al. [52] and evaluated by Madsen et al. [30], a label layout is frozen after optimization and updated when users change their viewpoint of the scene.

Optimization Criteria. We implemented several criteria that allow us to optimize the label placement. The optimization is based on an analysis of the user’s actual view through the HMD generated with EPR. We provide references for criteria that are derived from previous work.

Note that when looking for label placement candidates, our algorithm considers the entire area covered by a label instead of just individual pixels of the projected label position. Summed area tables are used to efficiently evaluate entire areas with respect to our criteria. For each criterion a threshold is used to remove candidate pixels that do not fulfill the criterion. See Fig. 7 for a visualization of the results and an overview of the algorithm.

4.3 Implementation

All optimization criteria are realized by analyzing the EPR views of the user’s eyes in order to identify good label placement candidates. Each pixel of the EPR view is a potential label position and represents the projected center of the label. Good placement candidates are pixels where the optimization criteria are met for all surrounding pixels that are covered by the respective label. Depending on its lightness, each label has its own set of placement candidates. To avoid overlapping labels the area covered by other labels is excluded from the set of possible label placement candidates. In our current implementation, we use a greedy strategy processing labels sequentially. After calculating the set of placement candidates for a label, we compute a vector field storing the direction towards the nearest candidate for each pixel. This vector field is used to guide labels towards positions that meet our optimization criteria.

Label placement candidates are initially computed in the EPR view of the dominant eye. However, to avoid stereo vision conflicts between the label content and the background, the Stereo Vision Uniformity criterion utilizes EPR views of both eyes. For this purpose, we project each candidate position of the dominant eye into the non-dominant eye. Then, we evaluate if the color of the corresponding candidate position is similar to the dominant eye. The algorithm only considers pixels of the EPR view of the dominant eye as good candidates where the absolute difference between the colors of the two EPR view pixels falls below a certain threshold. This straightforward stereo optimization strategy supplements the other optimization criteria that already take care of identifying uniform and untextured areas for label placement, and, thus, refines the label position to avoid stereo conflicts.

5.1 Eye Perspective Rendering

We performed an evaluation to explore the feasibility of the three EPR methods as well as their limitations for our eye-perspective view management solution.

Figure 8:

View Figure

Apparatus.. We created a dedicated capturing rig (Fig. 8 (Left)) consisting of two Stereo ZED Mini cameras. The top camera represented the built-in HMD camera capturing images I_{H, RGBD} and providing tracking poses using Stereo ZED’s native tracking, while the bottom camera captured data representing the user’s eyes E_i providing the ground truth I_EG for the synthesized EPR views I_E. The spatial arrangement of the cameras represented a typical HMD configuration. We based this configuration on the North Star HMD¹ that we also used in the subsequent user study. We calibrated the intrinsic and extrinsic camera using multi-camera calibration². The setup is connected to a standard PC. The implemented EPR methods run in real-time on an Intel Core i9-10900K, with 3.7 GHz, and an NVIDIA GeForce RTX 2080 Ti, 11GB. Note that similar approaches have been shown to also run on mobile devices [47, 54].

Scenes.. We used two AR scenes in our analysis, which vary in structural complexity in order to explore the methods for various target scenarios Fig. 9. We applied our approach to a simple scene consisting mainly of 2D planes (2D scene) and a more complex scene containing depth disparities (3D scene).

Data Collection.. Analogously to Fig. 3, we compared EPR views generated with the three render methods against a ground truth captured by a stereo camera representing the user’s eyes by calculating per-pixel quality differences of an optimization criterion utilizing ground truth or EPR information. For this evaluation, we utilized the quality criterion of background uniformity. Exemplary results are shown in Fig. 9.

Figure 9:

View Figure

Discussion.. As expected, the homography-based method provided reliable information only for mostly planar scenes for which an accurate plane can be detected. Naturally, any 3D geometry is distorted as seen in the color difference images leading to color and quality mismatches compared to the ground truth. A clear advantage of homography methods is that they can be realized with limited hardware resources as depth cameras are not necessarily required. While the planarity requirement is a clear limitation of the technique, homographies are feasible for use cases where content is applied to planar surfaces such as walls, e.g., when placing virtual windows, or also for distant scenes.

Figure 10:

View Figure

The reprojection method using only RGBD information of the user’s current viewpoint generally provided reliable information for view management for 2D and 3D scenes. Note that while we used a stereo-based RGBD camera in our setup, we relied only on the RGB image of one camera as typical HMDs contain only one color camera. Missing information after reprojecting data into the user’s view can be clearly identified and handled by the view management algorithm (see cyan pixels). Alternatively, an IBR-based inpainting strategy can be used to fill in missing information.

We noticed that the IBR-based method suffered from ghosting artifacts, which is a common issue of real-time IBR methods and has also been encountered by previous work [11]. These ghosting artifacts come from imprecise geometric proxies due to inaccuracies in the depth map as well as pose inaccuracies due to the incremental nature of SLAM pose tracking and its continuous pose optimizations (e.g., loop closure). To solve these issues, previous work, for instance, utilized precise poses and geometry from dedicated reconstruction algorithms that do not work in real-time [32] or utilized computationally complex algorithms that cannot be deployed to a mobile platform [58].

To strike a balance between reconstruction quality, flexibility regarding the complexity of scene structures, computational complexity, as well as performance, we decided to base our eye-perspective view management algorithm on the reprojection EPR method. This straightforward method allowed us to generate views of even complex 3D structures in real-time. However, a view management algorithm has to consider missing information due to disocclusions.

5.2 View Management

We designed a within-subject study to evaluate eye-perspective view management. We were interested if the view management would be able to successfully improve contrast, legibility, and avoid interferences with the background when reading labels. The study would also show, if sufficiently precise EPR can be realized for individual users wearing an HMD and changing their viewpoint.

Our independent variable was view management with three conditions (Fig. 10): (1) a baseline condition, where no view management was used to adjust label positions, (2) a condition where view management was based on the information captured by a built-in camera as commonly present in HMDs, and (3) a condition utilizing eye-perspective view management based on EPR to optimize label layouts based on the computed user’s actual view.

Previous work has evaluated the influence of text contrast and legibility on participants’ performance [8, 13]. Therefore, our study focuses on evaluating the ability of eye-perspective view management to enforce the optimization criteria that will result in better label layouts for the user’s view through an HMD in terms of contrast, legibility, and avoidance of interferences with the background. The study also explored shortcomings of view management relying only on a built-in HMD camera.

Participants. We recruited 12 participants (3 female, \(\overline{X}\) = 32.3 (7.7) years). On a scale from one to five (best), the mean of self-rated AR experience was 2.1 (sd = 1.4, median = 1.5). All participants had normal or corrected-to-normal vision and were not affected by color vision deficiency. We determined the dominant eye of participants with the Miles test [33] as input to the view management algorithm.

Apparatus. Participants performed the experiment in a seated position in front of a table standing in front of a wall. They were seated on a swivel chair that could easily move to change viewpoints during the tasks. The height of the chair was adjusted depending on the height of the users. The scene presented to the participants consisted of various color and texture patches. The back wall of the setup consisted of a 52 inch screen that was used for the initial instructions and example for participants, single-point active alignment method (SPAAM) calibration [55], and background scene elements. When designing the real-world scene for the user study, we took care to use simple geometric shapes in order to improve replicability of the experiment. Earlier studies showed that lightness contrast and not color contrast is the driving issue of perceptual conflicts with the background [61] and thus our labels were all light gray (RGB color: #B9B9B9, lightness 75). Three labels were placed in the scene. The baseline condition did not change the initial label position, while in the other two conditions, the labels were optimized by the respective view management method.

For our study, participants wore a Project North Star HMD with an attached Stereo ZED mini for tracking and RGBD depth data (Fig. 8 (Right)). The HMD and cameras were connected to a PC with Intel Core i9-10900K, 3.7 GHz, and an NVIDIA GeForce RTX 2080 Ti, 11GB. To replicate the semi-transparency of the HMD displays affecting the perceived scene colors, the colors of the EPR and HMD view for the respective conditions were modified by subtracting a constant value of RGB #1E0023 that was determined by minimizing the color difference between both views. The environmental light was set to 628.3 lux at a color temperature of 5604K using a LUPO Superpanel Dual Color 60 and measured via a Mavospec Base Spectrometer. The offset between RGBD camera and the user’s eyes was calibrated with SPAAM [19, 55] where participants had to align nine points at two different distances (approx. 70cm and 1m). The quality of the calibration was verified by the participants by overlaying the rendered EPR views using the HMD displays for each eye separately. Participants calibrated until the views were aligned. Furthermore, participants were shown a test label on the HMD at the same distance as the actual labels and asked to read it using both eyes, to ensure a good quality stereo calibration.

Task. Users had to align their viewpoint with a predefined position and orientation and evaluate the quality of the label placement with respect to the optimization criteria of the view management algorithm described in section 4.2. The forced viewpoint changes lead to updates of the layout to reflect the novel viewpoint. Once participants had aligned their viewpoints with the predefined viewpoint, participants were instructed to avoid changing the viewpoint to prevent further changes in the background. Furthermore, during analysis, each label was frozen in place to avoid unwanted layout updates due to involuntary small head motions.

Data Collection. For each label, participants rated the overall legibility and interference of the background using a 5-point rating scale, where 1 was the lowest rating. Participants also analyzed the layout of each label with respect to the utilized optimization criteria: (1) background uniformity, (2) lightness contrast, (3) texture contrast, and (4) stereo uniformity. For the analysis, we calculated the overall label quality for each label as the average of the four quality criteria. Participants also rated the legibility of labels (1 to 5, 5 best) and the negative influence of the background on legibility (1 to 5, 5 worst). After rating all labels for a viewpoint and view management condition, participants rated task difficulty using the Single Ease Question (SEQ) and mental effort using the Paas scale [41]. After finishing all three view management conditions for a viewpoint, participants were asked to rank the three strategies according to perceived layout quality. Participants rated the optimization criteria by estimating percentages in steps of 10 where higher percentages meant better quality. Background uniformity was rated by estimating the percentage occupied by the largest uniform area behind the label. Lightness contrast and texture contrast were rated by estimating the percentage of the label background that was not too bright or textured, respectively. Stereo uniformity was evaluated by estimating the percentage of uniform background between both eyes. This analysis was performed for both eyes separately. To avoid unnecessarily extending the user study, participants only performed this analysis for the dominant eye, when the background turned out to be uniform for both eyes.

Procedure.. Participants were recruited via university mailing lists and performed the study in a room on campus. After filling out an informed consent form and demographics questionnaire, participants were given a short introduction to the study. The introduction included a tutorial on SPAAM calibration to train participants to perform the procedure by themselves. To give participants an impression of various SPAAM calibration qualities, the experimenter showed them overlays filmed through the HMD that were recorded before the experiment. To make sure, participants understood the task and the involved analysis, they were also shown exemplary labels filmed through the HMD. Participants were then seated, put on the HMD, and calibrated their eye positions using SPAAM. Afterwards, the first view management method placed labels in the scene and participants assumed the first viewpoint. The experimenter queried the participants for each label and noted their answers. After rating all three labels for a viewpoint, the next view management condition was shown. After all view management conditions, participants had to rank the conditions from best to worst based on the layout quality. Afterward, participants assumed the next viewpoint and the procedure was repeated. In total, participants had to assume three different viewpoints leading to various backgrounds. An experiment lasted for approximately 90 minutes. The order of conditions and view positions was balanced using an orthogonal Latin Square table. We calculated the mean of each rating for each condition for each participant over all the viewpoints. With 3 (conditions) x 3 (labels) x 3 (viewpoints) = 27 repetitions for each participant and task, for 12 participants there were 324 trials.

Hypotheses.. We expected that (H1) world-camera view management and eye-perspective view management outperform the baseline condition (no view management) because even though optimizations that rely on the integrated HMD camera are not precise, labels are generally pushed towards more suitable areas. We expected that (H2) eye-perspective view management will outperform HMD camera view management because the EPR view replicates the user’s view and places labels in areas that the view management identified as suitable due to the optimization.

Results.. We analyzed the statistics as described in section 3. Results are summarized in Table 1 and  Table 2, box plots and ranking data are visualized in Figure 11. Participants ranked eye-perspective view management first in 97\(\%\) of all cases, participants ranked HMD-based view management second in 69\(\%\) of all cases, and baseline third in 69\(\%\) of all cases. In 16\(\%\) of all cases, participants could not decide to rank HMD or baseline method second or third. However, they ranked eye-perspective view management in the first place.

Figure 11:

View Figure

5.3 Discussion

In the following, we provide a summary of our learnings that can guide the use of eye-perspective view management algorithms.

Eye-Perspective View Management Improves Layouts.. Our user study clearly showed that eye-perspective view management using EPR can optimize label layouts for individual users wearing an OST HMD. Based on the statistically significant results of the user study, we partially accept H1 as eye-perspective view management outperformed the baseline layout. In addition, we accept H2 as eye-perspective view management outperformed HMD camera view management in all measured dependent variables. Overall, the average label layout quality, judged by participants was close to 100% for the EPR layout, compared to 78% for the HMD camera layout, and 75.5% for the baseline naive layout. As the view management took care of finding homogeneous areas for placing labels, the EPR-based label layout leads to significantly better legibility of labels and fewer distractions caused by the background. In 97% of all cases, participants preferred the EPR label layout, emphasizing the impact of layout optimizations from the user’s eye perspective.

Camera-Based Optimization Still Beneficial. We only partially accept H1 in terms of the HMD layout outperforming the baseline layout. The HMD layout outperformed the baseline layout in terms of uniformity and texture contrast. We did not find a significant difference in stereo quality for HMD and baseline, which is not surprising as both do not optimize the layout taking into account both eyes of the user. Interestingly, the HMD layout did not outperform the baseline in terms of legibility, background influence, or overall label quality. The baseline layout was even significantly better in terms of lightness contrast than the HMD layout. This may indicate that layouts based on wrong assumptions of the user’s view (e.g., HMD camera view), do not offer benefits compared to a very basic layout that is not optimized. However, more likely, this result can be explained by labels being placed in areas of the scene exhibiting different lightness characteristics. This placement may have adversely affected the overall label quality of the HMD view.

When exploring the lightness contrast issue, we noticed that during the study both HMD and baseline conditions placed labels in low lightness contrast areas. However, the HMD labels were overlapping with areas of comparably higher lightness. Due to the additive nature of the OST HMD display, these areas made it likely impossible for participants to read the labels, thereby, influencing the lightness contrast estimates. To explore this issue, we removed all label data that participants evaluated as having lightness contrast issues (90 of 324 labels), i.e., we filtered the data and kept only labels that were estimated as 100% good lightness contrast labels. While eye-perspective view management still outperformed both HMD and baseline layout, the HMD layout quality was significantly better than the baseline layout quality (HMD: 84.8 (17.6), 82.5; baseline: 78.3 (18.2), 81.25), than the baseline legibility (HMD: 3.6 (1.4), 4; baseline: 3.1 (1.6), 3) and than the baseline background influence (HMD: 2.4 (1.6), 2; baseline: 3.0 (1.6), 3). Note that we refrain from reporting results from statistical testing, as this analysis was not part of the initial hypotheses and should be investigated further.

In summary, optimization relying on the HMD camera may still be more beneficial than having no layout optimization in place. This is backed up by preference data, as the majority of participants (69% of all viewpoints) ranked HMD layout in second place and the baseline layout in third place (69% of all viewpoints).

EPR Replicates the User’s Views.. Our main study showed that EPR can replicate the user’s view through an OST HMD with sufficient precision so that eye-perspective optimization algorithms such as view management can adapt label layouts to improve legibility. More importantly, unlike previous studies utilizing OST displays [14, 27], participants were not forced to use a headrest but were allowed to change their view during the user study. Hence, we demonstrated that EPR can be utilized in mobile real-world scenarios which enables further exploration of effective view management for real-world use cases, such as a localized assembly task, or when users walk through a scene. EPR can also be applied beyond view management for other use cases for OST HMDs relying on real-world background information, such as content-aware interaction methods [39], or supporting people with visual impairments [34].

Choice of EPR Algorithms.. Based on our analysis of EPR algorithms, we decided to utilize the rather straightforward reprojection method as input to the eye-perspective view management. The reprojection method has the advantage that due to the low computational demand, it can run on mobile hardware and utilizes less battery power compared to more elaborate methods [58]. Furthermore, the reprojection method utilizes only the information from the current viewpoint of users and, therefore, can also react to scene changes. Therefore, the reprojection method is generally a good choice to explore the impact of EPR for various OST HMD use cases. However, missing information due to disocclusion artifacts may be a strong limiting factor of this method under certain conditions. Note that EPR is most useful for scenarios where a user is close to scene geometry. EPR may not be necessary when users view distant scenes as the parallax between camera and the user’s eyes does not introduce noticeable differences in both views [5].

Disocclusion Artifacts.. Disocclusion artifacts cause missing information in the synthesized eye-perspective view due to the offset between the user’s eyes and the camera providing color information of the scene. Disocclusion artifacts occur at the borders of 3D scene geometry and grow larger the closer the view is to the scene [5]. While our eye-perspective view management can handle disocclusion artifacts, it may fail to find valid placement areas for larger disocclusions when 3D geometry is too close to the user, e.g., when users pick up the geometry for closer inspection using their hands. Hence, advanced inpainting methods may be required to reconstruct the occluded scene [36, 37, 58], or algorithms may rely on alternative optimization methods.

Alternative Optimizations.. When reliable scene information is not available, e.g., due to disocclusion artifacts, or there is an insufficient number of valid locations, view management algorithms can utilize alternative forms of optimizations. Furthermore, while freezing label layouts [30, 52] may be an appropriate temporal coherence strategy for localized use cases, alternative strategies must be developed for scenarios where the real-world background constantly changes, e.g., while walking through a scene. Previous work has demonstrated that adapting the representation of a label using a uniform billboard color as label background can improve legibility, compensating for background interferences [8, 14]. However, such adaptations change the label design and may alter the meaning of information when using distinct color codes to communicate information such as security-critical details. Alternatively, label positions can be modified by sticking them to real-world scene geometry so that labels are at the same depth as the real-world background, thereby avoiding labels positioned in midair [40]. Such an approach also improves temporal coherence, as the label background will not change during viewpoint changes, and allows an optimization algorithm to utilize the view of the HMD camera as a reliable source of color information because the label background will be the same from the HMD camera’s view and the user’s eye view. However, sticking labels to real-world geometry in the background but annotating foreground objects will lead to a perceptual conflict as users have to switch between the depth of the annotated foreground object and the label positioned in the background.

Optimization as Loss Function.. In our user evaluation, we did not allow the eye-perspective view management tolerances, which forced labels to be placed in locations where all the constraints are fulfilled. Hence, we compared an optimal solution for label placement layout to the potentially worst outcome using imprecise scene information. HMD-camera-based layout and baseline layout inadvertently lead to layouts, where label quality deteriorated on average to below 80%. However, such a strict interpretation of the optimization criteria may lead to layouts where labels are placed at a large distance from their origin, or to situations where the scene structure does not allow the algorithm to find a layout solution. Instead of strictly enforcing the criteria, they can also be integrated into a loss function for a more lenient optimization that always finds a solution for label placement. Future work should investigate which criteria can be violated to which degree in order to achieve a layout with legible labels. This information can then be utilized to identify weights for the optimization criteria.