Optical See-through Augmented Reality Research Articles

Design of AR system tracking registration method using dynamic target light-field

In the process of tracking registration for an augmented reality (AR) system, it's essential first to obtain the system's initial state, as its accuracy significantly influences the precision of subsequent three-dimensional tracking registration. At this point, minor movements of the target can directly lead to calibration errors. Current methods fail to address the challenge of capturing the initial state of dynamic deformation in optically transparent AR systems effectively. To tackle this issue, the concept of a static light-field is expanded to a four-dimensional dynamic light-field, and a tracking registration method for an optical see-through AR system based on the four-dimensional dynamic light-field is introduced. This method begins by analyzing the relationship between the components of the optical see-through AR system and studying the impact of a dynamic target on the initial state model. Leveraging the fundamental principle of light-field correlation, the theory and model for four-dimensional dynamic light-field tracking registration are developed. A lot of experiments have confirmed the algorithm's accuracy, enhanced its stability, and demonstrated the superior performance of the three-dimensional tracking registration algorithm.

Optical see-through augmented reality fire safety training for building occupants

Fire safety training is crucial to increase building occupants' chances of surviving a fire emergency. Traditional training methods, such as lectures using video presentations, have limitations that can hinder learning performance. This article describes the development and testing of an alternative training solution using Augmented Reality (AR) technology. Through a controlled between-subject experiment, the AR-based fire safety training method was compared against a conventional video-based one based on participants' knowledge acquisition and retention, intrinsic motivation, and self-efficacy. Results suggest that the AR system was well-designed and as effective as the traditional method in terms of knowledge acquisition and retention and overall learning experience. However, it was found to be superior in terms of intrinsic motivation gain and self-efficacy retention. These findings demonstrate the potential of AR-based training methods to enhance building occupants' safety and provide directions for future developments and research in the field.

VoxAR: Adaptive Visualization of Volume Rendered Objects in Optical See-Through Augmented Reality.

We present VoxAR, a method to facilitate an effective visualization of volume-rendered objects in optical see-through head-mounted displays (OST-HMDs). The potential of augmented reality (AR) to integrate digital information into the physical world provides new opportunities for visualizing and interpreting scientific data. However, a limitation of OST-HMD technology is that rendered pixels of a virtual object can interfere with the colors of the real-world, making it challenging to perceive the augmented virtual information accurately. We address this challenge in a two-step approach. First, VoxAR determines an appropriate placement of the volume-rendered object in the real-world scene by evaluating a set of spatial and environmental objectives, managed as user-selected preferences and pre-defined constraints. We achieve a real-time solution by implementing the objectives using a GPU shader language. Next, VoxAR adjusts the colors of the input transfer function (TF) based on the real-world placement region. Specifically, we introduce a novel optimization method that adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. Finally, we present an assessment of our approach through objective evaluations and subjective user studies.

A Comparative Evaluation of Optical See-Through Augmented Reality in Surgical Guidance.

During traditional surgeries, planning and instrument guidance is displayed on an external screen. Recent developments of augmented reality (AR) techniques can overcome obstacles including hand-eye discoordination and heavy mental load. Among these AR technologies, optical see-through (OST) schemes with stereoscopic displays can provide depth perception and retain the physical scene for safety considerations. However, limitations still exist in certain AR systems and the influence of these factors on surgical performance is yet to explore. To this end, experiments of multi-scale surgical tasks were carried out to compare head-mounted display (HMD) AR and autostereoscopic image overlay (AIO) AR, concerning objective performance and subjective evaluation. To solely analyze effects brought by display techniques, the tracking system in each included display system was identical and similar tracking accuracy was proved by a preliminary experiment. Focus and context rendering was utilized to enhance in-situ visualization for surgical guidance. Latency values of all display systems were assessed and a delay experiment proved the latency differences had no significant impact on user performance. Results of multi-scale surgical tasks showed that HMD outperformed in detailed operations probably due to stable resolution along the depth axis, while AIO had better performance in larger-scale operations for better depth perception. This article helps point out the critical limitations of current OST AR techniques and potentially promotes the progress of AR applications in surgical guidance.

The Effect of Interocular Contrast Differences on the Appearance of Augmented Reality Imagery

Augmented reality (AR) devices seek to create compelling visual experiences that merge virtual imagery with the natural world. These devices often rely on wearable near-eye display systems that can optically overlay digital images to the left and right eyes of the user separately. Ideally, the two eyes should be shown images with minimal radiometric differences (e.g., the same overall luminance, contrast, and color in both eyes), but achieving this binocular equality can be challenging in wearable systems with stringent demands on weight and size. Basic vision research has shown that a spectrum of potentially detrimental perceptual effects can be elicited by imagery with radiometric differences between the eyes, but it is not clear whether and how these findings apply to the experience of modern AR devices. In this work, we first develop a testing paradigm for assessing multiple aspects of visual appearance at once, and characterize five key perceptual factors when participants viewed stimuli with interocular contrast differences. In a second experiment, we simulate optical see-through AR imagery using conventional desktop LCD monitors and use the same paradigm to evaluate the multi-faceted perceptual implications when the AR display luminance differs between the two eyes. We also include simulations of monocular AR systems (i.e., systems in which only one eye sees the displayed image). Our results suggest that interocular contrast differences can drive several potentially detrimental perceptual effects in binocular AR systems, such as binocular luster, rivalry, and spurious depth differences. In addition, monocular AR displays tend to have more artifacts than binocular displays with a large contrast difference in the two eyes. A better understanding of the range and likelihood of these perceptual phenomena can help inform design choices that support high-quality user experiences in AR.

Analysis of the Saliency of Color-Based Dichoptic Cues in Optical See-Through Augmented Reality.

In a future of pervasive augmented reality (AR), AR systems will need to be able to efficiently draw or guide the attention of the user to visual points of interest in their physical-virtual environment. Since AR imagery is overlaid on top of the user's view of their physical environment, these attention guidance techniques must not only compete with other virtual imagery, but also with distracting or attention-grabbing features in the user's physical environment. Because of the wide range of physical-virtual environments that pervasive AR users will find themselves in, it is difficult to design visual cues that "pop out" to the user without performing a visual analysis of the user's environment, and changing the appearance of the cue to stand out from its surroundings. In this article, we present an initial investigation into the potential uses of dichoptic visual cues for optical see-through AR displays, specifically cues that involve having a difference in hue, saturation, or value between the user's eyes. These types of cues have been shown to be preattentively processed by the user when presented on other stereoscopic displays, and may also be an effective method of drawing user attention on optical see-through AR displays. We present two user studies: one that evaluates the saliency of dichoptic visual cues on optical see-through displays, and one that evaluates their subjective qualities. Our results suggest that hue-based dichoptic cues or "Forbidden Colors" may be particularly effective for these purposes, achieving significantly lower error rates in a pop out task compared to value-based and saturation-based cues.

Invited Session IV: Extended reality--applications in vision science and beyond: Improving augmented reality through perceptual science.

Augmented reality (AR) systems make it possible to present visual stimuli that intermix and interact with people's view of the natural world. But building an AR system that merges stimuli with our natural visual experience is hard. AR systems often suffer from technical and visual limitations, such as small eyeboxes, limited brightness, and narrow visual field coverage. An integral part of AR system development, therefore, is perceptual research that improves our understanding of when and why these limitations matter. I will describe a suite of perceptual studies designed to provide guidance for engineers on the visibility and appearance of wearable optical see-through AR displays. Our results highlight the idiosyncrasies of how our visual system integrates information from the two eyes, the multifaceted nature of perceptual phenomena in AR, and the trade-offs that are currently necessary to create an AR system that is both wearable and compelling.

Fast and Robust Mid-Air Gesture Typing for AR Headsets using 3D Trajectory Decoding.

We present a fast mid-air gesture keyboard for head-mounted optical see-through augmented reality (OST AR) that supports users in articulating word patterns by merely moving their own physical index finger in relation to a virtual keyboard plane without a need to indirectly control a visual 2D cursor on a keyboard plane. To realize this, we introduce a novel decoding method that directly translates users' three-dimensional fingertip gestural trajectories into their intended text. We evaluate the efficacy of the system in three studies that investigate various design aspects, such as immediate efficacy, accelerated learning, and whether it is possible to maintain performance without providing visual feedback. We find that the new 3D trajectory decoding design results in significant improvements in entry rates while maintaining low error rates. In addition, we demonstrate that users can maintain their performance even without fingertip and gesture trace visualization.

Effects of Luminance Contrast on Depth Perception in Optical See-Through Augmented Reality

ABSTRACT Augmented reality (AR) technology presents the rendered virtual object in the real world whose depth cues can assist the depth estimation of the virtual object. For many AR applications, the successful interaction between the real world and the virtual object relies on the accuracy of depth perception. In the past, the research of depth perception in AR mainly focused on the impact of virtual object properties, including the size, color, and shading model. In this study, an experiment has been conducted to investigate the influence of ambient luminance, virtual object luminance, and shading model which is a luminance-related parameter on depth perception using an optical see-through head-mounted display. Results show that a higher luminance contrast between the real scene and the virtual object leads to a lower accuracy of depth perception in AR.

Supporting Tremor Rehabilitation Using Optical See-Through Augmented Reality Technology.

Tremor is a movement disorder that significantly impacts an individual's physical stability and quality of life, and conventional medication or surgery often falls short in providing a cure. Rehabilitation training is, therefore, used as an auxiliary method to mitigate the exacerbation of individual tremors. Video-based rehabilitation training is a form of therapy that allows patients to exercise at home, reducing pressure on rehabilitation institutions' resources. However, it has limitations in directly guiding and monitoring patients' rehabilitation, leading to an ineffective training effect. This study proposes a low-cost rehabilitation training system that utilizes optical see-through augmented reality (AR) technology to enable tremor patients to conduct rehabilitation training at home. The system provides one-on-one demonstration, posture guidance, and training progress monitoring to achieve an optimal training effect. To assess the system's effectiveness, we conducted experiments comparing the movement magnitudes of individuals with tremors in the proposed AR environment and video environment, while also comparing them with standard demonstrators. Participants wore a tremor simulation device during uncontrollable limb tremors, with tremor frequency and amplitude calibrated to typical tremor standards. The results showed that participants' limb movement magnitudes in the AR environment were significantly higher than those in the video environment, approaching the movement magnitudes of the standard demonstrators. Hence, it can be inferred that individuals receiving tremor rehabilitation in the AR environment experience better movement quality than those in the video environment. Furthermore, participant experience surveys revealed that the AR environment not only provided a sense of comfort, relaxation, and enjoyment but also effectively guided them throughout the rehabilitation process.

Tracking registration of optical see-through augmented reality based on the Riemannian manifold constraint.

Two three-dimensional tracking registration methods combined with Riemannian manifold object constraints are proposed to solve problems of low accuracy and instability of three-dimensional tracking registration in sparse and complex scenes. A deep convolution neural network is used to extract three-dimensional instance objects from the location by analyzing reasons that affect registration accuracy in sparse and complex scenes. The three-dimensional tracking registration model is established according to the Riemannian manifold constraint relationship of instance objects in different states. The stability of the three-dimensional tracking registration algorithm is improved by combining inertial sensors, and cumulative error is optimized using instance object labels to improve algorithm robustness. The proposed algorithm can effectively improve the accuracy of three-dimensional tracking registration. It can improve the performance of augmented reality systems and be applied to power system navigation, medical, and other fields.

Freehand Gestural Selection with Haptic Feedback in Wearable Optical See-Through Augmented Reality

Augmented reality (AR) technologies can blend digital and physical space and serve a variety of applications intuitively and effectively. Specifically, wearable AR enabled by optical see-through (OST) AR head-mounted displays (HMDs) might provide users with a direct view of the physical environment containing digital objects. Besides, users could directly interact with three-dimensional (3D) digital artefacts using freehand gestures captured by OST HMD sensors. However, as an emerging user interaction paradigm, freehand interaction with OST AR still requires further investigation to improve user performance and satisfaction. Thus, we conducted two studies to investigate various freehand selection design aspects in OST AR, including target placement, size, distance, position, and haptic feedback on the hand and body. The user evaluation results indicated that 40 cm might be an appropriate target distance for freehand gestural selection. A large target size might lower the selection time and error rate, and a small target size could minimise selection effort. The targets positioned in the centre are the easiest to select, while those in the corners require extra time and effort. Furthermore, we discovered that haptic feedback on the body could lead to high user preference and satisfaction. Based on the research findings, we conclude with design recommendations for effective and comfortable freehand gestural interaction in OST AR.

Shedding Light on Cast Shadows: An Investigation of Perceived Ground Contact in AR and VR.

Virtual objects in augmented reality (AR) often appear to float atop real world surfaces, which makes it difficult to determine where they are positioned in space. This is problematic as many applications for AR require accurate spatial perception. In the current study, we examine how the way we render cast shadows-which act as an important monocular depth cue for creating a sense of contact between an object and the surface beneath it-impacts spatial perception. Over two experiments, we evaluate people's sense of surface contact given both traditional and non-traditional shadow shading methods in optical see-through augmented reality (OST AR), video see-through augmented reality (VST AR), and virtual reality (VR) head-mounted displays. Our results provide evidence that nontraditional shading techniques for rendering shadows in AR displays may enhance the accuracy of one's perception of surface contact. This finding implies a possible tradeoff between photorealism and accuracy of depth perception, especially in OST AR displays. However, it also supports the use of more stylized graphics like non-traditional cast shadows to improve perception and interaction in AR applications.

Improving Naturalness in Transparent Augmented Reality with Image Gamma and Black Level

In this study, we used a custom-built Optical See-Through Augmented Reality (OST AR) system to conduct a psychophysical experiment to determine the preferred gamma and black level for high naturalness perception in OST-AR. We utilized 6 different fruit stimuli and 11 different backgrounds to do this experiment. We used two-way ANOVA to analyze the data and concluded that only the effect of different fruits on virtual objects’ gamma preference for high naturalness is considered statistically significant. Surprisingly, all ANOVA analyses indicate background’s color does not contribute to observers’ gamma or black level preference for naturalness. We found that gamma preference has a strong correlation with the average lightness of the virtual stimuli. There is no clear correlation between chroma, hue, and gamma preference in terms of naturalness perception. This finding suggests that the background can be ignored in future imaging pipelines emphasizing high naturalness appearance in Augmented Reality.

Optical see-through augmented reality can induce severe motion sickness

The aim was to investigate whether severe symptoms of visually induced motion sickness (VIMS) can occur in augmented reality (AR) optical see-through applications. VIMS has been extensively studied in virtual reality (VR), whereas it has received little attention in the context of AR technology, in which the real world is enhanced by virtual objects. AR optical see-through glasses are becoming increasingly popular as technology advances. Previous studies showed minor oculomotor symptoms of VIMS with the aforementioned technology. New applications with more dynamic simulations could alter previously observed symptom severity and patterns. In experiment 1, we exposed subjects to a traditional static AR application for pilot candidate training. In experiment 2, subjects completed tasks in a dynamic starfield simulation. We analyzed symptom profiles pre and post with the simulator sickness questionnaire (SSQ) and during exposure with the Fast Motion Sickness Scale (FMS). We also developed a new FMS-D that captures symptoms of dizziness during simulation. As expected, in experiment 1 we found low VIMS symptomatology with predominantly oculomotor symptoms. In experiment 2, on the other hand, we detected severe VIMS symptoms in some subjects, with disorientation (SSQ subscale) as the main symptom group. The present work demonstrates that VIMS can be of serious concern in modern AR applications. The FMS-D represents a new tool to measure symptoms of dizziness during exposure. VIMS symptoms need to be considered in the design and usage of future AR applications with dynamic virtual objects, e.g. for flight training or machine maintenance work.

Field-of-view correction in a VR HMD display with varifocal optics

Nowadays, we observe a progressive increase amount of virtual technologies application. VR devices, i.e. HMDs, are still struggling with the VAC. It is possible to “implement” the accommodation mechanism in HMDs using a varifocal system. By using focus tuneable lenses, we achieve high speed and smoothness of displacement of the virtual image plane. This solution involves also unfavourable changes of geometrical parameters of the projected image i.e. change in magnification. This can be solved by software modification parameters of the virtual camera. In this paper, the developed construction of varifocal HMD is presented, including the method of maintaining the constancy magnification. Full Text: PDF ReferencesY-J. Wang , Y-H. Lin, "Liquid crystal technology for vergenceaccommodation conflicts in augmented reality and virtual reality systems: a review", Liq. Cryst. Rev. 9. 35 (2021). CrossRef P.P. Monticone, M. Menozzi, M. Schlup, "Instrument for high-speed recording of accommodation of the human eye", Biomed. Tech. 55: 83 (2010). CrossRef G. Kramida, "Resolving the vergence-accommodation confict in headmounted displays", IEEE Trans. Vis. Comput. Graph. 22, 1912 (2016). CrossRef S. Shiwa, K. Omura, and F. Kishino, "Proposal for a 3-D display with accommodative compensation: 3DDAC," J. Soc. Inf. Disp. 4, 255 (1996). CrossRef K. Rathinavel, G. Wetzstein, H. Fuchs, "Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics," IEEE Trans. Vis. Comput. Graph. 25, 3125 (2019). CrossRef

The Effect of Context Switching, Focal Switching Distance, Binocular and Monocular Viewing, and Transient Focal Blur on Human Performance in Optical See-Through Augmented Reality.

In optical see-through augmented reality (AR), information is often distributed between real and virtual contexts, and often appears at different distances from the user. To integrate information, users must repeatedly switch context and change focal distance. If the user's task is conducted under time pressure, they may attempt to integrate information while their eye is still changing focal distance, a phenomenon we term transient focal blur. Previously, Gabbard, Mehra, and Swan (2018) examined these issues, using a text-based visual search task on a one-eye optical see-through AR display. This paper reports an experiment that partially replicates and extends this task on a custom-built AR Haploscope. The experiment examined the effects of context switching, focal switching distance, binocular and monocular viewing, and transient focal blur on task performance and eye fatigue. Context switching increased eye fatigue but did not decrease performance. Increasing focal switching distance increased eye fatigue and decreased performance. Monocular viewing also increased eye fatigue and decreased performance. The transient focal blur effect resulted in additional performance decrements, and is an addition to knowledge about AR user interface design issues.

Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

Use Case of the Application of the Optical-See-Through Augmented Reality Technology in the Engineering Context

Augmented Reality (AR) is the technology that enables the overlaying of physical objects with computer-generated virtual perceptible data in real time to provide an interactive user experience in the real environment. The potential of this technology to digitize and to improve the different engineering processes (e.g. manufacturing, logistics, maintenance, training) are apparent. Therefore, the application of AR in the engineering area has gained in importance in the recent years. This development can also be observed in the high-tech industry, companies like Microsoft, Google and Apple have intensified their efforts to develop new AR software and hardware. One of the important methods of AR is optical-see-through technology. The optical see-through AR enable the display of the virtual content on a transparent projection surface with the advantage that the users can at the same time perceive the real environment directly with their own eyes. This paper presents two use cases for the implementation of AR applications using the optical see-through technology and the HoloLens as an AR device. The first use case addresses the application of AR for Pick-by-Light assisted assembly process. The second use case deals with the use of AR for the implementation of a control application for robot shelving system. The experiences gathered during the performance of these use cases and the evaluation of the applicability of the optical see-through technology in the engineering field will be discussed in this paper. This presented work has an experimental character and rather than a fundamental research. The developed AR applications were implemented as a prototype by the students of the University of Applied Silences Karlsruhe during the AR laboratory. The gained knowledge in this work can be considered as suggestions for further research works.

P56 Early Health Technology Assessment to Guide Introduction of Novel Technology into the Market: The Vostars System Case Study

The aim of the study is to show potential approaches and usefulness of early Health Technology Assessment exemplifying the case study of the Video and Optical See-Through Augmented Reality Surgical System (VOSTARS), an advanced navigation tool for non-endoscopic surgeries, recently developed, also thanks to funding from H2020 programme.