Augmented Reality Display Research Articles

Understanding User Intention to Adopt Augmented Reality Head-Up Display: A Study Based on Design Factors

Augmented Reality Head-Up Display (AR-HUD) is designed to engender immersive experiences and possess vast potential in enhancing vehicular safety. In the context of advancing digital technologies, the intention to use such innovations represents a significant field of study. Drawing on the Perceived Value Theory and the Stimulus-Organism-Response (S-O-R) framework, this study formulates a model to evaluate user intentions regarding the use of automotive Augmented Reality Head-Up Display (AR-HUD). This model integrates design elements, including user interface attractiveness, sense of presence, controllability, and responsiveness, along with perceived utilitarian and hedonic values. Through Structural Equation Modeling (SEM) of 230 valid questionnaire results, the analysis identifies both utilitarian and hedonic values as pivotal in influencing AR-HUD adoption, with utilitarian value being preeminent. Additionally, it is evidenced that design attributes influence the usage intention via perceived utilitarian and hedonic values. Consequently, this study presents design strategies for auto manufacturers and designers aimed at amplifying the advantages of AR-HUD, thus fostering greater user adoption willingness.

Enhancing manual inspection in semiconductor manufacturing with integrated augmented reality solutions

On-site routine inspection often remains a manual operation in the semiconductor manufacturing industry because implementing automated solutions can be costly and technically challenging in such a highly controlled and complex environment. The manual inspection is prone to errors due to the impact of demanding physical and mental workloads. This paper presents an integrated Augmented Reality (AR) solution developed to assist manual inspection tasks in the supporting areas of semiconductor manufacturing, referred to as the sub-fab. The solution is accessible to a human worker wearing an AR headset during the inspection process at the location. We propose a system framework to deploy computational intelligences of varying granularity provided by the solution across cloud, edge, and device levels, accommodating constraints within the sub-fab. A machine maintenance module helps estimate and monitor the health condition of running scrubbers. Incorrect intentions performed by the worker on the scrubber control panel are detected through hand gesture recognition. This instantly prompts warning messages in the AR headset to prevent subsequent wrong actions. The solution can also identify abnormal device states through 6D pose estimation of objects enabled by machine learning models. A test scenario demonstrates how these functional features enhance the inspection efficiency and quality by reducing human workloads. This work demonstrates that semiconductor manufacturing may require AR-assisted functions different from those needed or common in other industrial sectors. It also highlights the potential of AR technology for reducing operational human errors in manual tasks.

Intraoperative Position System guided with Augmented Reality Improves the Learning Curve of Endovascular Navigation in Endovascular Naïve Operators

ObjectiveThis study aimed to compare the completion of gate cannulation task performed by participants of varying experience using fluoroscopy, the Intra-Operative Positioning System (IOPS) – an FDA-cleared endovascular navigation system that has been developed to reduce dependence on fluoroscopy – or an investigational augmented reality electromagnetic navigation technology based on IOPS. MethodsThe task consisted in the cannulation of the gate of a GORE Excluder AAA endoprosthesis bifurcated aortic stent graft (W.L. GORE & Associates, Flagstaff, AZ USA) deployed into a 3D printed abdominal aortic aneurysm (AAA) model connected to a 7000 MDX flow pump (Sarns Inc/3M, Ann Arbor, MI USA) reproducing physiological conditions. The procedure was performed in a hybrid operating room (GE Allia IGS 7). Each participant performed the cannulation task with fluoroscopy, standard IOPS guidance with flat screen display (IOPS-FS), and the investigational IOPS with augmented reality headset (IOPS-AR), in a randomly assigned order. All participants used the same sensorized guidewire and steerable 6Fr catheter during their three cannulation tasks. A total of 26 participants were classified in three groups of experience: Group 1 (endovascular naïve; n = 13), Group 2 (surgeon in training; n = 12) and Group 3 (one expert surgeon). Primary endpoints included cannulation time and technical success, which was defined as the advancement of the catheter over the guidewire within the main body of the aortic stent graft within a maximum 15 minutes cutoff time for each trial. ResultsIn group 1, the mean cannulation time was shorter using IOPS-AR versus fluoroscopy (4.3 ±4.4 min vs 7.1±4.9 min; P=.04), but not statistically different when comparing IOPS-FS and fluoroscopy (6.3±4.5 min vs 7.1±4.9 min; P=.63). In group 1, technical success was 77% with fluoroscopy and 92% with both IOPS-FS and IOPS-AR (P=.59). In group 2, although there was no significant difference between cannulation time amongst the three different endovascular approaches, there was a trend towards shorter cannulation times with IOPS-FS or IOPS-AR as compared to fluoroscopy (mean time of 2.5±0.9, 4.4±4.0 and 5.2±4.5 min, respectively). In group 2, technical success was 92% with fluoroscopy and 100% with both IOPS-FS and IOPS-AR (P>.99). The expert vascular surgeon repeated the cannulation task four times for each endovascular approach, with 100% technical success and no difference in mean cannulation time between the imaging modalities (P=.89). ConclusionsAugmented reality allows for reducing the gate cannulation time as compared to fluoroscopy in participants with no previous exposure to any endovascular procedure. This suggests that augmented reality can be beneficial for individuals early in their career and can mitigate the learning curve. As individuals become experts, their ability to adapt to different endovascular modalities increases, eliminating the learning curve altogether.

Efficient and Accurate Semi-Automatic Neuron Tracing with Extended Reality.

Neuron tracing, alternately referred to as neuron reconstruction, is the procedure for extracting the digital representation of the three-dimensional neuronal morphology from stacks of microscopic images. Achieving accurate neuron tracing is critical for profiling the neuroanatomical structure at single-cell level and analyzing the neuronal circuits and projections at whole-brain scale. However, the process often demands substantial human involvement and represents a nontrivial task. Conventional solutions towards neuron tracing often contend with challenges such as non-intuitive user interactions, suboptimal data generation throughput, and ambiguous visualization. In this paper, we introduce a novel method that leverages the power of extended reality (XR) for intuitive and progressive semi-automatic neuron tracing in real time. In our method, we have defined a set of interactors for controllable and efficient interactions for neuron tracing in an immersive environment. We have also developed a GPU-accelerated automatic tracing algorithm that can generate updated neuron reconstruction in real time. In addition, we have built a visualizer for fast and improved visual experience, particularly when working with both volumetric images and 3D objects. Our method has been successfully implemented with one virtual reality (VR) headset and one augmented reality (AR) headset with satisfying results achieved. We also conducted two user studies and proved the effectiveness of the interactors and the efficiency of our method in comparison with other approaches for neuron tracing.

Tasks Reflected in the Eyes: Egocentric Gaze-Aware Visual Task Type Recognition in Virtual Reality.

With eye tracking finding widespread utility in augmented reality and virtual reality headsets, eye gaze has the potential to recognize users' visual tasks and adaptively adjust virtual content displays, thereby enhancing the intelligence of these headsets. However, current studies on visual task recognition often focus on scene-specific tasks, like copying tasks for office environments, which lack applicability to new scenarios, e.g., museums. In this paper, we propose four scene-agnostic task types for facilitating task type recognition across a broader range of scenarios. We present a new dataset that includes eye and head movement data recorded from 20 participants while they engaged in four task types across 15 360-degree VR videos. Using this dataset, we propose an egocentric gaze-aware task type recognition method, TRCLP, which achieves promising results. Additionally, we illustrate the practical applications of task type recognition with three examples. Our work offers valuable insights for content developers in designing task-aware intelligent applications. Our dataset and source code are available at zhimin-wang.github.io/TaskTypeRecognition.html.

Artificial intelligence guidance for 3D Augmented Reality Robotic surgery: when the machine falls the human assistance is still alive

Artificial intelligence guidance for 3D Augmented Reality Robotic surgery: when the machine falls the human assistance is still alive

Study of the Dispersion Compensation Double-Layer Diffractive Optical Components Based on Metasurface and Grating, and Their Application in Augmented Reality Displays.

We employed a double-layer coupled diffractive optical element, based on metasurfaces and diffraction gratings, which exhibits wavefront modulation and chromatic dispersion compensation. Utilizing this double-layer coupled diffractive optical element in the optical information transmission process of a diffractive waveguide allows for the transmission of color image information using a single-layer waveguide structure. Our results demonstrate that, under the conditions of a field of view of 47° × 47°, an entrance pupil size of 2.9 × 2.9 mm2, and an exit pupil extension size of 8.9 mm, the uniformity of the brightness for each monochromatic field reached 85%, while the uniformity of color transmission efficiency exceeded 95%.

Lights, Headset, Tablet, Action: Exploring the Use of Hybrid User Interfaces for Immersive Situated Analytics

While augmented reality (AR) headsets provide entirely new ways of seeing and interacting with data, traditional computing devices can play a symbiotic role when used in conjunction with AR as a hybrid user interface. A promising use case for this setup is situated analytics. AR can provide embedded views that are integrated with their physical referents, and a separate device such as a tablet can provide a familiar situated overview of the entire dataset being examined. While prior work has explored similar setups, we sought to understand how people perceive and make use of visualizations presented on both embedded visualizations (in AR) and situated visualizations (on a tablet) to achieve their own goals. To this end, we conducted an exploratory study using a scenario and task familiar to most: adjusting light levels in a smart home based on personal preference and energy usage. In a prototype that simulates AR in virtual reality, embedded visualizations are positioned next to lights distributed across an apartment, and situated visualizations are provided on a handheld tablet. We observed and interviewed 19 participants using the prototype. Participants were easily able to perform the task, though the extent the visualizations were used during the task varied, with some making decisions based on the data and others only on their own preferences. Our findings also suggest the two distinct roles that situated and embedded visualizations can have, and how this clear separation might improve user satisfaction and minimize attention-switching overheads in this hybrid user interface setup. We conclude by discussing the importance of considering the user's needs, goals, and the physical environment for designing and evaluating effective situated analytics applications.

Comparative Analysis of AR-HUDs Crash Warning Icon Designs: An Eye-Tracking Study Using 360° Panoramic Driving Simulation

Augmented Reality Head-Up Displays (AR-HUDs) enhance driver perception and safety, yet optimal hazard warning design remains unclear. This study examines three AR-HUD crash warning icon types (BD, BR, BW) across various turning scenarios. Using a 360-degree video-based driving simulation with 36 participants, eye-tracking metrics were collected. Results show BW icons, dynamically linked to hazards, significantly improve drivers’ pedestrian risk awareness and visual attention allocation compared to BD and BR systems. BW consistently demonstrated longer gaze duration, higher fixation counts, and shorter time to first fixation across all turns. BD and BR icons were more susceptible to lane changes, potentially diverting attention from hazards. Findings suggest prioritizing dynamic tracking warning icons over fixed-position alternatives to minimize visual competition and distraction, providing crucial insights for AR-HUD optimization in automated vehicles.

A larger augmented-reality field of view improves interaction with nearby holographic objects.

Augmented-reality (AR) applications have shown potential for assisting and modulating gait in health-related fields, like AR cueing of foot-placement locations in people with Parkinson's disease. However, the size of the AR field of view (AR-FOV), which is smaller than one's own FOV, might affect interaction with nearby floor-based holographic objects. The study's primary objective was to evaluate the effect of AR-FOV size on the required head orientations for viewing and interacting with real-world and holographic floor-based objects during standstill and walking conditions. Secondary, we evaluated the effect of AR-FOV size on gait speed when interacting with real-world and holographic objects. Sixteen healthy middle-aged adults participated in two experiments wearing HoloLens 1 and 2 AR headsets that differ in AR-FOV size. To confirm participants' perceived differences in AR-FOV size, we examined the head orientations required for viewing nearby and far objects from a standstill position (Experiment 1). In Experiment 2, we examined the effect of AR-FOV size on head orientations and gait speeds for negotiating 2D and 3D objects during walking. Less downward head orientation was required for looking at nearby holographic objects with HoloLens 2 than with HoloLens 1, as expected given differences in perceived AR-FOV size (Experiment 1). In Experiment 2, a greater downward head orientation was observed for interacting with holographic objects compared to real-world objects, but again less so for HoloLens 2 than HoloLens 1 along the line of progression. Participants walked slightly but significantly slower when interacting with holographic objects compared to real-world objects, without any differences between the HoloLenses. To conclude, the increased size of the AR-FOV did not affect gait speed, but resulted in more real-world-like head orientations for seeing and picking up task-relevant information when interacting with floor-based holographic objects, improving the potential efficacy of AR cueing applications.

The Fuse Platform: Integrating data from IoT and other Sensors into an Industrial Spatial Digital Twin

Abstract. Digital Twins as virtual representations of industrial assets are being used to assimilate varied sources of data for improved awareness and decision making in operations and process optimisation. This paper explores the integration of IoT sensors into a spatial digital twin called Fuse that Woodside Energy has been building for the assets it operates. We describe the Fuse platform and its knowledge graph data core that is used to organise and inter-relate data for presentation within 3D visualisations, domain-specific contexts and immersive augmented reality presentations. The key contribution here is the use of a knowledge graph to link diverse data sources so as to contextualise sensor data for actionable insights. One area of significant innovation has been the development and use of new Internet of Things (IoT) devices which have been enabled by advances in sensor technology, connectivity, and cloud computing. These new tailored data sources are complimenting existing plant and resource planning data for improved asset monitoring, predictive maintenance and process automation.

Nested Semi-Transparent Isosurface Simulated Volume-Rendering (NESTIS-VR) – An efficient on-device rendering approach for Augmented Reality headsets increasing surgeon confidence of kidney donor arterial anatomy

Background and objectiveVolume-renderings of computed tomography or magnetic resonance angiograms (MRAs) are routinely used by surgeons in the preoperative assessment of vascular anatomy in kidney donors. Stereoscopic headsets (OST-HMD) like Microsoft HoloLens allow intuitive interaction with three-dimensional content for more intuitive comprehension, but do not allow real-time ray-casting volume-rendering of medical volume datasets on-device due to computational limitations. MethodsWe introduce NEsted Semi-Transparent Isosurface Simulated Volume-Rendering (NESTIS-VR), as an on-device alternative to ray-casting volume-rendering and developed an application for HoloLens to render kidney donor MRAs with interactive control of fundamental rendering parameters. We compared NESTIS-VR with current standard pre-calculated 2D ray-cast volume-renderings in an observational study with 2 expert kidney transplant surgeons, measuring their confidence in pre-operatively assessing the kidney pedicle arterial anatomy in 20 potential donors. We also compared it against other 3D rendering techniques to understand which features contributed most to any improvements. ResultsReal-time stereoscopic three-dimensional (3D) NESTIS-VR in Augmented Reality significantly improves surgeons’ confidence compared with pre-calculated conventional two-dimensional (2D) ray-casting volume-rendered images (p = 0.0415/p = 0.00003). 2D non-stereoscopic NESTIS-VR was significantly superior to pre-calculated 2D ray-casting volume-rendered images for both surgeons (p = 0.044/p = 0.0003). Single isosurface 2D rendering was significantly superior than pre-calculated 2D volume-rendered images for one surgeon. There was no significant difference between binocular 3D display over 2D views with NESTIS-VR or between constrained and unconstrained vantage points for 2D viewing. ConclusionNESTIS-VR provides a new approach to rendering medical datasets in computationally limited OST-HMD headsets and significantly increases surgeons’ confidence of kidney donor arterial anatomy. The principal confidence benefit arises from providing surgeons interactive control over rendering parameters compared to pre-calculated renderings at preset parameters whilst rendering on-device and keeping the OST-HMD untethered from a workstation.

Obstacle negotiation while dual-tasking in children with Developmental Coordination Disorder (DCD): An augmented-reality approach

BackgroundChildren with Developmental Coordination Disorder (DCD) exhibit deficits in predictive motor control, balance, and aspects of cognitive control, which are important for safely negotiating obstacles while walking. As concurrent performance of cognitive and motor tasks (dual-tasking) may exacerbate these deficits, we examined motor and cognitive dual-tasking differences between children with DCD and their typically developing (TD) peers during obstacle negotiation. Methods34 children aged 6–12 years (16 TD, 18 DCD) walked along a 12 m path, stepping over an obstacle (30 % or 50 % of leg length) at its mid-point. On dual-task trials, participants completed a simple or complex (cognitive) visual discrimination task presented via an augmented reality headset. Proportional dual-task costs (pDTCs) were measured on cognitive and gait outcomes over three phases: pre-obstacle, obstacle step-over, and post-obstacle. ResultsDuring the obstacle step-over phase, both groups increased their leading leg clearance when dual-tasking, while the DCD group had larger pDTC than TD for the high obstacle under simple stimulus conditions (viz simple-high combination). The complex cognitive task produced larger pDTCs than the simple one on leading leg clearance and post-obstacle gait variability. ConclusionsIn general, both DCD and TD groups showed similar pDTCs under complex conditions, while the specific deficit in DCD under the simple-high combination suggests a (default) compensatory strategy during step-over when attention is diverted to a secondary task. Competing cognitive and motor demands during obstacle negotiation present a potential safety risk for children.

Design of freeform elements with a large exit pupil for AR display

This work presents an augmented reality display consisting of two freeform lenses. The first lens is composed of two aspherical polished surfaces and one plane surface, while the second lens acts as a compensator to allow distortion-free viewing of the surrounding area. One benefit of this design is the large size of the exit pupil, reaching up to 12mm, and the low distortion value of 1.63%. The optics has a thickness of 22mm and is compatible with 0.49-inch diagonal displays. The field of view is limited to 28◦, which is sufficient for various augmented reality applications.

An Interactive Vision-Based 3D Augmented Reality System for In-Home Physical Rehabilitation: A Qualitative Inquiry to Inform System Development.

Postoperative physical rehabilitation is crucial after total joint replacement (TJR). However, completing the recommended levels of postoperative physical exercise is challenging for many older adults with TJR. Lack of adequate postoperative physical exercise has negative consequences on rehabilitation outcomes. Innovative rehabilitation tools for postoperative physical exercises are needed to ensure successful rehabilitation outcomes among older adults with TJR. The aim of this study is to explore key knowledge users' perspectives about how to design an interactive vision-based three-dimensional augmented reality system (3D ARS) to support in-home postoperative physical rehabilitation for older adults with TJR. We conducted a qualitative descriptive study involving 11 semi-structured interviews and six focus groups with 42 older adults with TJR and four unrelated family caregivers. Data were analysed using thematic analysis. Participant insights were grouped into two main themes: (1) dreaming up possibilities and (2) being pragmatic. The first theme captured participants' reflections on the potential utility of a 3D ARS for postoperative physical rehabilitation and features that could be embedded in the 3D ARS to support successful postoperative physical rehabilitation. The second theme captured participants' reflections on practical issues and considerations that could impact access and usage of the 3D ARS. These findings provide researchers, rehabilitation providers and system developers with the foundations for designing, implementing and evaluating innovative augmented reality tools that support effective in-home physical rehabilitation among older adults with TJR. Research users (i.e., individuals and organisations invested in and using the research findings) were actively engaged throughout this work. Specifically, a meeting was held between the research team and representatives of an Expert by Experience team (individuals with lived experience), which was established to support the National Research Council's (organisation) Aging in Place programme. During this meeting, the idea to develop and evaluate an ARS for postoperative physical rehabilitation of older adults with TJR was supported. Research users had the opportunity to review the current study protocol and provide feedback on the study design, offering direction to maximize the relevance and usefulness of our findings to the National Research Council Canada's Aging in Place programme. Research users contributed to participant recruitment efforts and the development of the interview guide. Two Experts by Experience also agreed to be on the Advisory Panel for this multi-phased study, supporting active engagement and centring the voice of research users in knowledge creation and implementation. These experts reviewed a brief report of the current study findings, and continue to guide how the study findings are used to inform the next phase of this multi-phased research.

Movement and positioning of a virtual cuboid in 3d space in an augmented reality environment

Human-virtual object interaction is common in both entertainment and work settings. This study investigated the movement time (MT) and subjective rating of difficulty (SRD) for moving a virtual cuboid in a 3D space. The participants wore an augmented reality (AR) headset, picked up a virtual cuboid, and placed it on an assigned target. They rated the SRD of the task on a five-point scale. The effects of the 3D coordinate of the target, sex, and handedness on the MT were analysed. The error placement rate was also recorded. Significant effects of spatial coordinates were found on both MT and SRD. Both single- and two-stage MT modelling were conducted using segmented and unsegmented MT data, respectively. The insignificant prediction error between the models indicates that the two-stage MT model is not superior to the single-stage one. The findings of this study are beneficial to software designers in designing user-friendly AR applications.

The Effect of Dynamic Effects and Color Transparency of AR-HUD Navigation Graphics on Driving Behavior Regarding Inattentional Blindness

In-vehicle Augmented Reality Head-Up Displays (AR-HUDs) offer drivers integrated navigation and vehicle status information, thereby improving the driving experience. Many studies explored the effects of AR-HUD navigation design and information presentation on driving performance; however, these studies neglected the occurrence of inattentional blindness, which may lead to potential safety hazards. This study used a two-factor within-subjects design to investigate how dynamic effects (pulsation, blinking, movement) and color transparency levels (T1, T0.75, T0.5, T0.35) of AR-HUD navigation graphics influence inattentional blindness and user experience. The results indicated that color transparency significantly affects the drivers’ inattentional blindness behavior and user experience. Specifically, T0.75 resulted in the lowest inattentional blindness, while T0.5 was associated with the shortest reaction time (without blindness) and highest user satisfaction. Dynamic effects did not significantly impact inattentional blindness, but blinking reduced reaction time (without blindness). This study provides design insights for vehicle AR-HUD navigation interfaces.

Implantation of Reverse Shoulder Endoprothesis Using Navigation.

The implantation of a reverse shoulder arthroplasty (RSA) is a proven method for pain relief and improvement in shoulder function. Results vary, depending on the patient's age, the severity of the disease, and the experience of the surgeon. Indications for RSA are diverse, ranging from rotator cuff arthropathy to fractures. Recent studies show improved survival rates and reduced complications after primary implantation. Preoperative planning using 3D-CT or MRI is considered to be the gold standard. Patient-specific instruments (PSI) have been introduced, but are associated with costs and waiting times. Navigation with augmented reality (AR) provides a more efficient alternative. The intraoperative transfer of the plan to the patient is carried out via AR glasses, allowing real-time information without having to divert the surgeon's attention from the surgical site. This optimises the workflow and potentially yields more precise implantation results. In summary, the combination of 3D planning, navigation, and AR offers a promising method for precise and efficient RSA-implantations. Nevertheless, long-term results and functional scores are not yet available.

Peptide Bundlemer Networks or Lattices: Controlling Cross-Linking and Self-Assembly Using Protein-like Display of Chemistry.

Coiled-coil 'bundlemer' peptides were selectively modified with allyloxycarbonyl (alloc)-protected lysine, a non-natural amino acid containing an alkene on its side chain. The specific display of this alkene from the coiled-coil surface with protein-like specificity enabled this residue to be used as a covalent linkage for creating peptide networks with controllable properties or as a physical linkage for the self-assembly of bundlemers into unexpected, intricate lattices driven by the hydrophobic nature of the side chain. For network formation, peptides were modified with both alloc-protected lysine and cysteine amino acids for solution assembly into solvent-swollen films and subsequent covalent cross-linking via thiol-ene photo click reactions. The degree of network cross-linking, as determined by rheometry, was finely tuned by varying the specific spatial display of reactive groups on the bundlemer building block particles, transitioning between intrabundle and interbundle cross-linking. The designed display of alloc groups from the center of the bundlemer building block also prompted particle self-assembly into an unexpected intricate lattice with a porous morphology. The lattices were studied in a variety of solution conditions using transmission electron microscopy, cryotransmission electron microscopy, and small-angle X-ray scattering. The approximate particle arrangement in the lattice was determined by using coarse-grained modeling and machine learning optimization techniques along with experimental methods. The proposed truss-like face-centered cubic packing of the alloc-functionalized bundlemers agrees well with the experimental results.

Compact binocular holographic near-eye 3D display system based on a liquid crystal polarization grating

Holographic technology is considered as one of the most ideal 3D display technologies and has significant potential in near-eye display. However, binocular holographic near-eye 3D display still faces challenges such as serious visual fatigue and complex system. Here, we propose a compact binocular holographic near-eye 3D display system based on a liquid crystal polarization grating (LCPG). Based on the LCPG, the reconstructed light can be deflected respectively to left and right viewing areas. The proposed system achieves binocular display with only one phase-only spatial light modulator used to load computer-generated hologram, which makes the binocular display system more compact. Simultaneously, the proposed system reduces mismatches in the images perceived by two eyes, thus achieving a better binocular display effect. Experimental results demonstrate the feasibility of the proposed binocular holographic near-eye 3D display system with dual viewing areas. Additionally, based on the time-division multiplexing method, dynamic binocular holographic near-eye display can be achieved. The proposed system has potential applications in holographic virtual reality and augmented reality display.