See-through Head-mounted Display Research Articles

Optical see-through head-mounted display including transmittance-variable display for high visibility

This paper presents an optical see-through head-mounted display (OST-HMD) that can provide a highly immersive augmented reality (AR) environment. Because visibility of objects is degraded by external light and because projection optics cannot represent black color, the reality of AR content is reduced. To solve this problem, we adopted an additional transparent display to adjust the transmittance of exterior light. This adjustment is achieved using a guest–host liquid crystal (GHLC) to provide transparency in the initial state and opacity when voltage is applied. To verify the display’s usefulness, we measured ambient contrast ratio (ACR) and color gamut under various ambient light conditions after attaching the GHLC panel to the existing OST-HMD (Microsoft HoloLens). Under typical office illumination, use of the GHLC panel increased the ACR by 4.67 times, and the color gamut by 2.87 times compared to the OST-HMD without it. Under high illumination the panel increased the ACR by 1.54 times and the color gamut by 16.16 times. The proposed method significantly improved the ambient contrast ratio and color expression, and yielded a flexible way to customize existing OST-HMDs by adding a detachable display.

A CNN-based multi-target fast classification method for AR-SSVEP

Because an augmented-reality-based brain-computer interface (AR-BCI) is easily disturbed by external factors, the traditional electroencephalograph (EEG) classification algorithms fail to meet the real-time processing requirements with a large number of stimulus targets or in a real environment. We propose a multi-target fast classification method for augmented-reality-based steady-state visual evoked potential (AR-SSVEP), using a convolutional neural network (CNN). To explore the availability and accuracy of high-efficiency multi-target classification methods in AR-SSVEP with a short stimulation duration, a similar stimulus layout was used for a computer screen (PC) and an optical see-through head-mounted display (OST-HMD) device (HoloLens). The experiment included nine flicker stimuli of different frequencies, and a multi-target fast classification method based on a CNN was constructed to complete nine classification tasks, for which the average accuracy of AR-BCI in our CNN model at 0.5- and 1-s stimulus duration was 67.93% and 80.83%, respectively. These results verified the efficacy of the proposed model for processing multi-target classification in AR-BCI.

Focal surface occlusion.

This paper proposes focal surface occlusion to provide focal cues of occlusion masks for multiple virtual objects at continuous depths in an occlusion-capable optical see-through head-mounted display. A phase-only spatial light modulator (PSLM) that acts as a dynamic free-form lens is used to conform the focal surface of an occlusion mask to the geometry of the virtual scene. To reproduce multiple and continuous focal blurs while reducing the distortion of the see-through view, an optical design based on afocal optics and edge-based optimization to exploit a property of the occlusion mask is established. The prototype with the PSLM and transmissive liquid crystal display can reproduce the focus blur of occluded objects at multiple and continuous depths with a field of view of 14.6°.

Towards Indistinguishable Augmented Reality

Adding virtual information that is indistinguishable from reality has been a long-awaited goal in Augmented Reality (AR). While already demonstrated in the 1960s, only recently have Optical See-Through Head-Mounted Displays (OST-HMDs) seen a reemergence, partially thanks to large investments from industry, and are now considered to be the ultimate hardware for augmenting our visual perception. In this article, we provide a thorough review of state-of-the-art OST-HMD-related techniques that are relevant to realize the aim of an AR interface almost indistinguishable from reality. In this work, we have an initial look at human perception to define requirements and goals for implementing such an interface. We follow up by identifying three key challenges for building an OST-HMD-based AR interface that is indistinguishable from reality: spatial realism, temporal realism, and visual realism. We discuss existing works that aim to overcome these challenges while also reflecting against the goal set by human perception. Finally, we give an outlook into promising research directions and expectations for the years to come.

An Extended Analysis on the Benefits of Dark Mode User Interfaces in Optical See-Through Head-Mounted Displays

Light-on-dark color schemes, so-called “Dark Mode,” are becoming more and more popular over a wide range of display technologies and application fields. Many people who have to look at computer screens for hours at a time, such as computer programmers and computer graphics artists, indicate a preference for switching colors on a computer screen from dark text on a light background to light text on a dark background due to perceived advantages related to visual comfort and acuity, specifically when working in low-light environments. In this article, we investigate the effects of dark mode color schemes in the field of optical see-through head-mounted displays (OST-HMDs), where the characteristic “additive” light model implies that bright graphics are visible but dark graphics are transparent . We describe two human-subject studies in which we evaluated a normal and inverted color mode in front of different physical backgrounds and different lighting conditions. Our results indicate that dark mode graphics displayed on the HoloLens have significant benefits for visual acuity and usability, while user preferences depend largely on the lighting in the physical environment. We discuss the implications of these effects on user interfaces and applications.

The Impact of Focus and Context Visualization Techniques on Depth Perception in Optical See-Through Head-Mounted Displays.

Estimating the depth of virtual content has proven to be a challenging task in Augmented Reality (AR) applications. Existing studies have shown that the visual system makes use of multiple depth cues to infer the distance of objects, occlusion being one of the most important ones. The ability to generate appropriate occlusions becomes particularly important for AR applications that require the visualization of augmented objects placed below a real surface. Examples of these applications are medical scenarios in which the visualization of anatomical information needs to be observed within the patient's body. In this regard, existing works have proposed several focus and context (F+C) approaches to aid users in visualizing this content using Video See-Through (VST) Head-Mounted Displays (HMDs). However, the implementation of these approaches in Optical See-Through (OST) HMDs remains an open question due to the additive characteristics of the display technology. In this article, we, for the first time, design and conduct a user study that compares depth estimation between VST and OST HMDs using existing in-situ visualization methods. Our results show that these visualizations cannot be directly transferred to OST displays without increasing error in depth perception tasks. To tackle this gap, we perform a structured decomposition of the visual properties of AR F+C methods to find best-performing combinations. We propose the use of chromatic shadows and hatching approaches transferred from computer graphics. In a second study, we perform a factorized analysis of these combinations, showing that varying the shading type and using colored shadows can lead to better depth estimation when using OST HMDs.

Enhancing edge indicator for visual field loss compensation for homonymous hemianopia patients

Homonymous hemianopia (HH) is a visual field defect in which the same side of both eyes is blind. In this paper, we propose an out-of-visual-field information compensation system for HH patients using an optical see-through head-mounted display. In the proposed system, in order to reduce the obstruction in the remaining visual field, indicators are placed at the boundary between the remaining visual field and the defective part to notify the HH patient about the events occurring at the blinded side of the visual field. Two different kinds of indicators, namely brightness-based and flicker-based, are designed to visualize the spatial information of the events. We conducted experiments with simulated HH participants to verify the effectiveness of the proposed system and compare the two designed indicators, as well as an existing method that makes use of the overlay overview window. The experimental results proved that the proposed system can react more intuitively and is faster than the existing method in the part near the central visual field. Moreover, it can alleviate the obstruction problem of the existing method.

Design of a Pupil-Matched Occlusion-Capable Optical See-Through Wearable Display.

State-of-the-art optical see-through head-mounted displays (OST-HMD) for augmented reality applications lack the ability to correctly render light blocking behavior between digital and physical objects, known as mutual occlusion capability. In this article, we present a novel optical architecture for enabling a high performance, occlusion-capable optical see-through head-mounted display (OCOST-HMD). The design utilizes a single-layer, double-pass architecture, creating a compact OCOST-HMD that is capable of rendering per-pixel mutual occlusion, correctly pupil-matched viewing perspective between virtual and real scenes, and a wide see-through field of view (FOV). Based on this architecture, we present a design embodiment and a compact prototype implementation. The prototype demonstrates a virtual display with an FOV of 34° by 22°, an angular resolution of 1.06 arc minutes per pixel, and an average image contrast greater than 40 percent at the Nyquist frequency of 53 cycles/mm. Furthermore, the device achieves a see-through FOV of 90° by 50°, within which about 40° diagonally is occlusion-enabled, and has an angular resolution of 1.0 arc minutes (comparable to a 20/20 vision) and a dynamic range greater than 100:1. We conclude the paper with a quantitative comparison of the key optical performance such as modulation transfer function, image contrast, and color rendering accuracy of our OCOST-HMD system with and without occlusion enabled for various lighting environments.

Augmented reality based navigation for distal interlocking of intramedullary nails utilizing Microsoft HoloLens 2

Background and objectiveThe distal interlocking of intramedullary nail remains a technically demanding procedure. Existing augmented reality based solutions still suffer from hand-eye coordination problem, prolonged operation time, and inadequate resolution. In this study, an augmented reality based navigation system for distal interlocking of intramedullary nail is developed using Microsoft HoloLens 2, the state-of-the-art optical see-through head-mounted display. MethodsA customized registration cube is designed to assist surgeons with better depth perception when performing registration procedures. During drilling, surgeons can obtain accurate and in-situ visualization of intramedullary nail and drilling path, and dynamic navigation is enabled. An intraoperative warning system is proposed to provide intuitive feedback of real-time deviations and electromagnetic disturbances. ResultsThe preclinical phantom experiment showed that the reprojection errors along the X, Y, and Z axes were 1.55 ± 0.27 mm, 1.71 ± 0.40 mm, and 2.84 ± 0.78 mm, respectively. The end-to-end evaluation method indicated the distance error was 1.61 ± 0.44 mm, and the 3D angle error was 1.46 ± 0.46°. A cadaver experiment was also conducted to evaluate the feasibility of the system. ConclusionOur system has potential advantages over the 2D-screen based navigation system and the pointing device based navigation system in terms of accuracy and time consumption, and has tremendous application prospects.

Teleproctoring for Neurovascular Procedures: Demonstration of Concept Using Optical See-Through Head-Mounted Display, Interactive Mixed Reality, and Virtual Space Sharing-A Critical Need Highlighted by the COVID-19 Pandemic.

Physician training and onsite proctoring are critical for safely introducing new biomedical devices, a process that has been disrupted by the pandemic. A teleproctoring concept using optical see-through head-mounted displays with a proctor's ability to see and, more important, virtually interact in the operator's visual field is presented. Test conditions were created for simulated proctoring using a bifurcation aneurysm flow model for WEB device deployment. The operator in the angiography suite wore a Magic Leap-1 optical see-through head-mounted display to livestream his or her FOV to a proctor's computer in an adjacent building. A Web-based application (Spatial) was used for the proctor to virtually interact in the operator's visual space. Tested elements included the quality of the livestream, communication, and the proctor's ability to interact in the operator's environment using mixed reality. A hotspot and a Wi-Fi-based network were tested. The operator successfully livestreamed the angiography room environment and his FOV of the monitor to the remotely located proctor. The proctor communicated and guided the operator through the procedure over the optical see-through head-mounted displays, a process that was repeated several times. The proctor used mixed reality and virtual space sharing to successfully project images, annotations, and data in the operator's FOV for highlighting any device or procedural aspects. The livestream latency was 0.71 (SD, 0.03) seconds for Wi-Fi and 0.86 (SD, 0.3) seconds for the hotspot (P = .02). The livestream quality was subjectively better over the Wi-Fi. New technologies using head-mounted displays and virtual space sharing could offer solutions applicable to remote proctoring in the neurointerventional space.

Representing Virtual Transparent Objects on Optical See-Through Head-Mounted Displays Based on Human Vision

In this study, we propose two methods for representing virtual transparent objects convincingly on an optical see-through head-mounted display without the use of an attenuation function or shielding environmental light. The first method represents the shadows and caustics of virtual transparent objects as illusionary images. Using this illusion-based approach, shadows can be represented without blocking the luminance produced by the real environment, and caustics are represented by adding the luminance of the environment to the produced shadow. In the second method, the visual effects that occur in each individual image of a transparent object are represented as surface, refraction, and reflection images by considering human binocular movement. The visual effects produced by this method reflect the disparities among the vergence and defocus of accommodation associated with the respective images. When reproducing the disparity, each parallax image is calculated in real time using a polygon-based method, whereas when reproducing the defocus, image processing is applied to blur each image and consider the user’s gaze image. To validate these approaches, we conducted experiments to evaluate the realism of the virtual transparent objects produced by each method. The results revealed that both methods produced virtual transparent objects with improved realism.

Augmented reality visualization for aiding biopsy procedure according to computed tomography based virtual plan

The purpose of this study was to develop and verify an intraoperative module for supporting navigated biopsy procedures using optical see-through head-mounted display (HMD). Biopsy procedure including entry and endpoints of needle insertion was planned preoperatively having regard to the resection region segmentation and safety margin definition. Biopsy procedures were performed by two users using an intraoperative optical navigation module on a specially prepared brain phantom. Two visualization techniques were compared: an accurate augmented reality one, where a virtual plan is superimposed onto surgical field by using optical see-through HMD together with personalized calibration method and visualization on the external display. Averaged errors from 24 trials using external display were 2.04 ± 0.83 mm for the first user and 2.69 ± 1.11 mm for the second one, while applying HMD 2.50 ± 0.93 mm (the first user) and 2.17 ± 0.82 mm (the second user), respectively. Proper usage of HMD visualization preceded by the personalized calibration allows the user to perform navigated biopsy procedure with comparable accuracy to its equivalent with the external display. Additionally, augmented reality visualization improves ergonomics and enables focusing on the surgical field without losing a direct line of sight with the field of view as it happens for external displays. However, ensuring high accuracy of augmented reality visualization still requires proper calibration and some user experience, which is challenging.

Design and Evaluation of Optical See-Through Head-Mounted Display With Wide FOV Based on Dihedral Corner Reflector Array

Existing optical see-through (OST) head-mounted displays (HMDs) generally utilize plane mirrors with partial transmission to merge the virtual and real-world scenes, which inducing the vignetting effect to result in small field of view (FOV). Accordingly, this paper proposes an OST-HMD system in which a rotationally-symmetric eyepiece with a wide FOV of ±30 degrees is coupled with a dihedral corner reflector array (DCRA) to achieve a sufficient relief distance to project all of the exit rays of the eyepiece into the pupil. The feasibility of the proposed system is evaluated by means of numerical simulations based on optical models of the HMD lens, DCRA and human eye. The simulation results show that the eye can receive the display information in the wide FOV with a sufficient relief distance through this kind of the OST-HMD development.

Optical See-Through Head-Mounted Displays With Short Focal Distance: Conditions for Mitigating Parallax-Related Registration Error.

Optical see-through (OST) augmented reality head-mounted displays are quickly emerging as a key asset in several application fields but their ability to profitably assist high precision activities in the peripersonal space is still sub-optimal due to the calibration procedure required to properly model the user's viewpoint through the see-through display. In this work, we demonstrate the beneficial impact, on the parallax-related AR misregistration, of the use of optical see-through displays whose optical engines collimate the computer-generated image at a depth close to the fixation point of the user in the peripersonal space. To estimate the projection parameters of the OST display for a generic viewpoint position, our strategy relies on a dedicated parameterization of the virtual rendering camera based on a calibration routine that exploits photogrammetry techniques. We model the registration error due to the viewpoint shift and we validate it on an OST display with short focal distance. The results of the tests demonstrate that with our strategy the parallax-related registration error is submillimetric provided that the scene under observation stays within a suitable view volume that falls in a ±10 cm depth range around the focal plane of the display. This finding will pave the way to the development of new multi-focal models of OST HMDs specifically conceived to aid high-precision manual tasks in the peripersonal space.

Augmented Reality Procedure Assistance System for Operator Training and Simulation

This study explores the design, implementation, and evaluation of an Augmented Reality (AR) prototype that assists novice operators in performing procedural tasks in simulator environments. The prototype uses an optical see-through head-mounted display (OST HMD) in conjunction with a simulator display to supplement sequences of interactive visual and attention-guiding cues to the operator’s field of view. We used a 2x2 within-subject design to test two conditions: with/without AR-cues, each condition had a voice assistant and two procedural tasks (preflight and landing). An experiment examined twenty-six novice operators. The results demonstrated that augmented reality had benefits in terms of improved situation awareness and accuracy, however, it yielded longer task completion time by creating a speed-accuracy trade-off effect in favour of accuracy. No significant effect on mental workload is found. The results suggest that augmented reality systems have the potential to be used by a wider audience of operators.

AR-Loupe: Magnified Augmented Reality by Combining an Optical See-Through Head-Mounted Display and a Loupe.

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

Comparison of Augmented Reality Display Techniques to Support Medical Needle Insertion.

Augmented reality (AR) may be a useful technique to overcome issues of conventionally used navigation systems supporting medical needle insertions, like increased mental workload and complicated hand-eye coordination. Previous research primarily focused on the development of AR navigation systems designed for specific displaying devices, but differences between employed methods have not been investigated before. To this end, a user study involving a needle insertion task was conducted comparing different AR display techniques with a monitor-based approach as baseline condition for the visualization of navigation information. A video see-through stationary display, an optical see-through head-mounted display and a spatial AR projector-camera-system were investigated in this comparison. Results suggest advantages of using projected navigation information in terms of lower task completion time, lower angular deviation and affirmative subjective participant feedback. Techniques requiring the intermediate view on screens, i.e. the stationary display and the baseline condition, showed less favorable results. Thus, benefits of providing AR navigation information compared to a conventionally used method could be identified. Significant objective measures results, as well as an identification of advantages and disadvantages of individual display techniques contribute to the development and design of improved needle navigation systems.

Focus-Adjustable Head Mounted Display with Off-Axis System

An off-axis system refers to an optical system in which the optical axis and the normal vector at the vertex of each surface do not match. An off-axis optical system can be applied in order to construct a thin and light optical system. In particular, the optical system used for a see-through head-mounted display (HMD) must be designed asymmetrically, with respect to the optical axis. Because the vision of a human is different for each individual, HMD requires focus adjustment. The effective focal length (EFL) of the optical system must be calculated to obtain the focus adjustment. However, the off-axis optical system cannot be calculated by conventional methods. In this study, the EFL was calculated by rotating the coordinates of the rays near the optical axis by the angle of reflection or refraction at the intersection of each surface, with the rays coinciding with the optical axis. The magnitude of movement of the micro-display for focus adjustment was obtained from the calculated EFL, for a see-through type HMD.

Optical module design for augmented reality glasses

Subject of Research. The paper considers an optical module design method for augmented reality glasses. The module contains a translucent beam-splitting element that provides observation of real objects with superimposed additional virtual image (OST HMD — optical see-through head-mounted display). The central element of the optical module is a prism that views two channels simultaneously: a real world picture and a virtual image. As a result, the user is able to see an augmented reality image. The functional scheme of the optical module with the introduced eye tracking system is considered. Method. Optimization of the prism surfaces, as well as tilts and relative positions, was performed using Zemax OpticStudio. It is based on the idea of applying free-form surfaces, which enables the sizes to be reduced, the field of view to be increased and the image quality to be improved. Main Results. The initial parameters of the optical element and an algorithm for optimization of free-form surfaces are developed, that gives the possibility to obtain a relatively wide field of vision (54° diagonally), compactness and high image quality parameters. Practical Relevance. The results of this work can be used in the design and development of augmented reality glasses in various fields, such as: medicine, online education, defense industry, sports, and marketing.

Fast and accurate online calibration of optical see-through head-mounted display for AR-based surgical navigation using Microsoft HoloLens.

The use of an optical see-through head-mounted display (OST-HMD) in augmented reality (AR) has significantly increased in recent years, but the alignment between the virtual scene and physical reality is still a challenge. A fast and accurate calibration method of OST-HMD is important for augmented reality in the medical field. We proposed a fast online calibration procedure for OST-HMD with the aid of an optical tracking system. Two 3D datasets are collected in this procedure: the virtual points rendered in front of the observer's eyes and the corresponding points in optical tracking space. The transformation between these two 3D coordinates is solved to build the connection between virtual and real space. An AR-based surgical navigation system is developed with the help of this procedure, which is used for experiment verification and clinical trial. Phantom experiment based on the 3D-printed skull is performed, and the average root-mean-square error of control points between rendered object and skull model is [Formula: see text]mm, and the time consumption of the calibration procedure can be less than 30s. A clinical trial is also conducted to show the feasibility in real surgery theatre. The proposed calibration method does not rely on the camera of the OST-HMD and is very easy to operate. Phantom experiment and clinical case demonstrated the feasibility of our AR-based surgical navigation system and indicated it has the potential in clinical application.