Augmented Reality Head-mounted Display Research Articles

Head-Mounted Augmented Reality Platform for Markerless Orthopaedic Navigation.

Visual augmented reality (AR) has the potential to improve the accuracy, efficiency and reproducibility of computer-assisted orthopaedic surgery (CAOS). AR Head-mounted displays (HMDs) further allow non-eye-shift target observation and egocentric view. Recently, a markerless tracking and registration (MTR) algorithm was proposed to avoid the artificial markers that are conventionally pinned into the target anatomy for tracking, as their use prolongs surgical workflow, introduces human-induced errors, and necessitates additional surgical invasion in patients. However, such an MTR-based method has neither been explored for surgical applications nor integrated into current AR HMDs, making the ergonomic HMD-based markerless AR CAOS navigation hard to achieve. To these aims, we present a versatile, device-agnostic and accurate HMD-based AR platform. Our software platform, supporting both video see-through (VST) and optical see-through (OST) modes, integrates two proposed fast calibration procedures using a specially designed calibration tool. According to the camera-based evaluation, our AR platform achieves a display error of 6.31 ± 2.55 arcmin for VST and 7.72 ± 3.73 arcmin for OST. A proof-of-concept markerless surgical navigation system to assist in femoral bone drilling was then developed based on the platform and Microsoft HoloLens 1. According to the user study, both VST and OST markerless navigation systems are reliable, with the OST system providing the best usability. The measured navigation error is 4.90 ± 1.04 mm, 5.96 ± 2.22 ° for the VST system, and 4.36 ± 0.80 mm, 5.65 ± 1.42 ° for the OST system.

Evaluating the integration of body donor imaging into anatomical dissection using augmented reality.

Augmented reality (AR) has recently been utilized as an integrative teaching tool in medical curricula given its ability to view virtual objects while interacting with the physical environment. The evidence for AR in medical training, however, is limited. For this reason, the purpose of this mixed method study was to evaluate the implementation of overlaying donor-specific diagnostic imaging (DSDI) onto corresponding body donors in a fourth-year, dissection-based, medical elective course entitled anatomy for surgeons (AFS). Students registered in AFS course were separated into groups, receiving either DSDI displayed on Microsoft HoloLens AR head-mounted display (n=12) or DSDI displayed on iPad (n=15). To test for the change in spatial ability, students completed an anatomical mental rotation test (AMRT) prior to and following the AFS course. Students also participated in a focus group discussion and completed a survey at the end of AFS, analyzed through thematic triangulation and an unpaired, Mann Whitney U test respectively, both addressing dissection experience, DSDI relevancy to dissection, and use of AR in anatomical education. Although statistically significant differences were not found when comparing student group AMRT scores, survey and discussion data suggest that the HoloLens had improved the students' understanding of, and their spatial orientation of, anatomical relationships. Trunk dissection quality grades were significantly higher with students using the HoloLens. Although students mentioned difficulties with HoloLens software, with faculty assistance, training, and enhanced software development, there is potential for this AR tool to contribute to improved dissection quality and an immersive learning experience.

Augmented Reality for Machine Setups: Task Performance and Usability Evaluation in a Field Test

Head-mounted displays (HMDs) for augmented reality (AR) have received much scholarly attention. Most studies on AR HMDs, however, rely on lab-based environments or simplistic tasks that do not reflect the complexity of industrial work processes. In this work, we conducted a field study in which professional machine operators performed a setup task for an injection molding machine. We assessed and compared task performance, perceived workload, and usability of an AR HMD versus a tablet computer. We find that the use of an AR HMD is perceived as positive overall and not mentally demanding. However, task completion times were not reduced, although participants felt they were.

Impact of information display on worker performance for wood frame wall assembly using AR HMD under different task conditions

Augmented Reality (AR) head-mounted displays (HMDs) provide users with an immersive virtual experience in the real world. The portability of this technology affords various information display options for construction workers that are not possible otherwise. However, the impact of these different information presentation options on human performance should be carefully evaluated before such technology is deployed in the jobsite. In this paper, we describe a research effort examining how different information displays presented via AR HMD influence task performance when assembling three sized wooden wall frame assembly tasks. We asked 18 construction engineering students with framing experience to finish three wood frame assembly tasks (large, medium, and small) using one of the three information displays (AR 3D conformal, AR 2D tag-along, and paper blueprints). The task performance was measured by time of completion and framing errors, which were analyzed and compared among each factor.

In Situ Visualization for 3D Ultrasound-Guided Interventions with Augmented Reality Headset.

Augmented Reality (AR) headsets have become the most ergonomic and efficient visualization devices to support complex manual tasks performed under direct vision. Their ability to provide hands-free interaction with the augmented scene makes them perfect for manual procedures such as surgery. This study demonstrates the reliability of an AR head-mounted display (HMD), conceived for surgical guidance, in navigating in-depth high-precision manual tasks guided by a 3D ultrasound imaging system. The integration between the AR visualization system and the ultrasound imaging system provides the surgeon with real-time intra-operative information on unexposed soft tissues that are spatially registered with the surrounding anatomic structures. The efficacy of the AR guiding system was quantitatively assessed with an in vitro study simulating a biopsy intervention aimed at determining the level of accuracy achievable. In the experiments, 10 subjects were asked to perform the biopsy on four spherical lesions of decreasing sizes (10, 7, 5, and 3 mm). The experimental results showed that 80% of the subjects were able to successfully perform the biopsy on the 5 mm lesion, with a 2.5 mm system accuracy. The results confirmed that the proposed integrated system can be used for navigation during in-depth high-precision manual tasks.

Real-time navigation guidance with intraoperative CT imaging for pedicle screw placement using an augmented reality head-mounted display: a proof-of-concept study.

Augmented reality (AR) has the potential to improve the accuracy and efficiency of instrumentation placement in spinal fusion surgery, increasing patient safety and outcomes, optimizing ergonomics in the surgical suite, and ultimately lowering procedural costs. The authors sought to describe the use of a commercial prototype Spine AR platform (SpineAR) that provides a commercial AR head-mounted display (ARHMD) user interface for navigation-guided spine surgery incorporating real-time navigation images from intraoperative imaging with a 3D-reconstructed model in the surgeon's field of view, and to assess screw placement accuracy via this method. Pedicle screw placement accuracy was assessed and compared with literature-reported data of the freehand (FH) technique. Accuracy with SpineAR was also compared between participants of varying spine surgical experience. Eleven operators without prior experience with AR-assisted pedicle screw placement took part in the study: 5 attending neurosurgeons and 6 trainees (1 neurosurgical fellow, 1 senior orthopedic resident, 3 neurosurgical residents, and 1 medical student). Commercially available 3D-printed lumbar spine models were utilized as surrogates of human anatomy. Among the operators, a total of 192 screws were instrumented bilaterally from L2-5 using SpineAR in 24 lumbar spine models. All but one trainee also inserted 8 screws using the FH method. In addition to accuracy scoring using the Gertzbein-Robbins grading scale, axial trajectory was assessed, and user feedback on experience with SpineAR was collected. Based on the Gertzbein-Robbins grading scale, the overall screw placement accuracy using SpineAR among all users was 98.4% (192 screws). Accuracy for attendings and trainees was 99.1% (112 screws) and 97.5% (80 screws), respectively. Accuracy rates were higher compared with literature-reported lumbar screw placement accuracy using FH for attendings (99.1% vs 94.32%; p = 0.0212) and all users (98.4% vs 94.32%; p = 0.0099). The percentage of total inserted screws with a minimum of 5° medial angulation was 100%. No differences were observed between attendings and trainees or between the two methods. User feedback on SpineAR was generally positive. Screw placement was feasible and accurate using SpineAR, an ARHMD platform with real-time navigation guidance that provided a favorable surgeon-user experience.

Augmented reality head-mounted display-based incision planning in cranial neurosurgery: a prospective pilot study.

Monitor and wand-based neuronavigation stations (MWBNSs) for frameless intraoperative neuronavigation are routinely used in cranial neurosurgery. However, they are temporally and spatially cumbersome; the OR must be arranged around the MWBNS, at least one hand must be used to manipulate the MWBNS wand (interrupting a bimanual surgical technique), and the surgical workflow is interrupted as the surgeon stops to "check the navigation" on a remote monitor. Thus, there is need for continuous, real-time, hands-free, neuronavigation solutions. Augmented reality (AR) is poised to streamline these issues. The authors present the first reported prospective pilot study investigating the feasibility of using the OpenSight application with an AR head-mounted display to map out the borders of tumors in patients undergoing elective craniotomy for tumor resection, and to compare the degree of correspondence with MWBNS tracing. Eleven consecutive patients undergoing elective craniotomy for brain tumor resection were prospectively identified and underwent circumferential tumor border tracing at the time of incision planning by a surgeon wearing HoloLens AR glasses running the commercially available OpenSight application registered to the patient and preoperative MRI. Then, the same patient underwent circumferential tumor border tracing using the StealthStation S8 MWBNS. Postoperatively, both tumor border tracings were compared by two blinded board-certified neurosurgeons and rated as having an excellent, adequate, or poor correspondence degree based on a subjective sense of the overlap. Objective overlap area measurements were also determined. Eleven patients undergoing craniotomy were included in the study. Five patient procedures were rated as having an excellent correspondence degree, 5 had an adequate correspondence degree, and 1 had poor correspondence. Both raters agreed on the rating in all cases. AR tracing was possible in all cases. In this small pilot study, the authors found that AR was implementable in the workflow of a neurosurgery OR, and was a feasible method of preoperative tumor border identification for incision planning. Future studies are needed to identify strategies to improve and optimize AR accuracy.

Immersive reading in virtual and augmented reality environment

Purpose This study aims to investigate “immersive reading,” which occurs when individuals read text while in a virtual reality (VR) or augmented reality (AR) environment. Design/methodology/approach In Experiment 1, 64 participants read text passages and answered multiple-choice questions in VR and AR head-mounted displays (HMDs) compared with doing the same task on liquid crystal display (LCD). In Experiment 2, 31 participants performed the same reading tasks but with two VR HMDs of different display quality. Findings Compared with reading on LCD as the baseline, participants reading in VR and AR HMDs got 82% (VR) and 88% (AR) of the information accurately. Participants tended to respond more accurately and faster, though not statistically significant, with the VR HMD of higher pixel density in the speed-reading task. Originality/value The authors observed the speed and accuracy of reading in VR and AR environments, compared with the reading speed and accuracy on an LCD monitor. The authors also compared the reading performance on two VR HMDs that differed in display quality but were otherwise similar in every way.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented reality in robotic assisted orthopaedic surgery: A pilot study

BackgroundThe research and development of augmented-reality (AR) technologies in surgical applications has seen an evolution of the traditional user-interfaces (UI) utilised by clinicians when conducting robot-assisted orthopaedic surgeries. The typical UI for such systems relies on surgeons managing 3D medical imaging data in the 2D space of a touchscreen monitor, located away from the operating site. Conversely, AR can provide a composite view overlaying the real surgical scene with co-located virtual holographic representations of medical data, leading to a more immersive and intuitive operator experience. Materials and MethodsThis work explores the integration of AR within an orthopaedic setting by capturing and replicating the UI of an existing surgical robot within an AR head-mounted display worn by the clinician. The resulting mixed-reality workflow enabled users to simultaneously view the operating-site and real-time holographic operating informatics when carrying out a robot-assisted patellofemoral-arthroplasty (PFA). Ten surgeons were recruited to test the impact of the AR system on procedure completion time and operating surface roughness. Results and DiscussionThe integration of AR did not appear to require subjects to significantly alter their surgical techniques, which was demonstrated by non-significant changes to the study’s clinical metrics, with a statistically insignificant mean increase in operating time (+0.778 s, p = 0.488) and a statistically insignificant change in mean surface roughness (p = 0.274). Additionally, a post-operative survey indicated a positive consensus on the usability of the AR system without incurring noticeable physical distress such as eyestrain or fatigue. ConclusionsOverall, these study results demonstrated a successful integration of AR technologies within the framework of an existing robot-assisted surgical platform with no significant negative effects in two quantitative metrics of surgical performance, and a positive outcome relating to user-centric and ergonomic evaluation criteria.

Augmented reality simulator for CT-guided interventions

IntroductionCT-guided interventions are taught using a mentored approach on real patients. It is well established that simulation is a valuable training tool in medicine. This project assessed the feasibility and acceptance of replicating a CT-guided intervention using a bespoke software application with an augmented reality head-mounted display (ARHMD).MethodsA virtual patient was generated using a CT dataset obtained from The Cancer Imaging Archive. A surface mesh of a virtual patient was projected into the field-of-view of the operator. ChArUco markers, placed on both the needle and agar jelly phantom, were tracked using RGB cameras built into the ARHMD. A virtual CT slice simulating the needle position was generated on voice command. The application was trialled by senior interventional radiologists and trainee radiologists with a structured questionnaire evaluating face validity and technical aspects.ResultsSixteen users trialled the application and feedback was received from all. Eleven felt the accuracy and realism was adequate for training and twelve felt more confident about their CT biopsy skills after this training session.DiscussionThe study showed the feasibility of simulating a CT-guided procedure with augmented reality and that this could be used as a training tool.Key Points• Simulating a CT-guided procedure using augmented reality is possible.• The simulator developed could be an effective training tool for clinical practical skills.• Complexity of cases can be tailored to address the training level demands.

Mitigating the Costs of Spatial Transformations With a Situation Awareness Augmented Reality Display: Assistance for the Joint Terminal Attack Controller 3-17.

Evaluate and model the advantage of a situation awareness (SA) supported by an augmented reality (AR) display for the ground-based joint terminal attack Controller (JTAC), in judging and describing the spatial relations between objects in a hostile zone. The accurate world-referenced description of relative locations of surface objects, when viewed from an oblique slant angle (aircraft, observation post) is hindered by (1) the compression of the visual scene, amplified at a lower slang angle, (2) the need for mental rotation, when viewed from a non-northerly orientation. Participants viewed a virtual reality (VR)-simulated four-object scene from either of two slant angles, at each of four compass orientations, either unaided, or aided by an AR head-mounted display (AR-HMD), depicting the scene from a top-down (avoiding compression) and north-up (avoiding mental rotation) perspective. They described the geographical layout of four objects within the display. Compared with the control condition, that condition supported by the north-up SA display shortened the description time, particularly on non-northerly orientations (9 s, 30% benefit), and improved the accuracy of description, particularly for the more compressed scene (lower slant angle), as fit by a simple computational model. The SA display provides large, significant benefits to this critical phase of ground-air communications in managing an attack-as predicted by the task analysis of the JTAC. Results impact the design of the AR-HMD to support combat ground-air communications and illustrate the magnitude by which basic cognitive principles "scale up" to realistically simulated real-world tasks such as search and rescue.

First Contact – Take 2: Using XR technology as a bridge between Māori, Pākehā and people from other cultures in Aotearoa, New Zealand

The art installation common/room explores human–digital–human encounter across cultural differences. It comprises a suite of extended reality (XR) experiences that use technology as a bridge to help support human connections with a view to overcoming intercultural discomfort (racism). The installations are exhibited as an informal dining room, where each table hosts a distinct experience designed to bring people together in a playful yet meaningful way. Each experience uses different technologies, including 360° 3D virtual reality (VR) in a headset (common/place), 180° 3D projection (Common Sense) and augmented reality (AR) (Come to the Table! and First Contact – Take 2). This article focuses on the latter, First Contact – Take 2, in which visitors are invited to sit at a dining table, wear an AR head-mounted display and encounter a recorded volumetric representation of an Indigenous Māori woman seated opposite them. She speaks directly to the visitor out of a culture that has refined collective endeavour and relational psychology over millennia. The contextual and methodological framework for this research is international commons scholarship and practice that sits within a set of relationships outlined by the Mātike Mai Report on constitutional transformation for Aotearoa, New Zealand. The goal is to practise and build new relationships between Māori and Tauiwi, including Pākehā.

Abstract No. 593 Preprocedural evaluation of ablation zone coverage utilizing a novel augmented reality platform

Earlier work has demonstrated bench and clinical usability of a novel augmented reality head-mounted display (AR HMD) platform developed for guidance in percutaneous thermal ablation (PTA). Platform functionality includes holographic projection of 3D modeled ablation zones based on ablation probe manufacturer specifications. The purpose of this study is to further evaluate the utility of this functionality for preprocedural planning to determine number and type of ablation probes needed for adequate tumor coverage.

Optical distortion correction considering radial and tangential distortion rates defined by optical design

This paper proposes a high-accuracy distortion correction method for optical imaging distortion mapping and computational image predistortion. Due to the establishment of the predistortion is based on the distortion evaluation in the common optical design software CODE V, it can be used in the broad field of optical imaging system. Two distortion correction methods considering radial and tangential distortion rates which defined by optical design are introduced to perform distortion correction, and the forward and inverse image warping interpolation methods are adopted, separately. Simulations are conducted to verify the effectiveness in correcting complex distortion for freeform off-axis optical system. Furthermore, we implement a distortion correction experiment for an augmented reality head-mounted display that suffers from an asymmetric distortion with the maximum tangential distortion over 7%. For the first method, fast and high-accuracy image warping is achieved, and the RMS error of the approximation predistortion image is less than 1 pixel. Meanwhile, the RMS errors of the corrected imaging points are approximately 1.93 pixels vertically and 2.16 pixels horizontally when using a 5-megapixel camera. The second method considers the additional influence of the actual machining and assembly errors of the optical system, and the RMS errors of 0.89 pixels vertically and 0.85 pixels horizontally are achieved.

Holographic patient tracking after bed movement for augmented reality neuronavigation using a head-mounted display

BackgroundHolographic neuronavigation has several potential advantages compared to conventional neuronavigation systems. We present the first report of a holographic neuronavigation system with patient-to-image registration and patient tracking with a reference array using an augmented reality head-mounted display (AR-HMD).MethodsThree patients undergoing an intracranial neurosurgical procedure were included in this pilot study. The relevant anatomy was first segmented in 3D and then uploaded as holographic scene in our custom neuronavigation software. Registration was performed using point-based matching using anatomical landmarks. We measured the fiducial registration error (FRE) as the outcome measure for registration accuracy. A custom-made reference array with QR codes was integrated in the neurosurgical setup and used for patient tracking after bed movement.ResultsSix registrations were performed with a mean FRE of 8.5 mm. Patient tracking was achieved with no visual difference between the registration before and after movement.ConclusionsThis first report shows a proof of principle of intraoperative patient tracking using a standalone holographic neuronavigation system. The navigation accuracy should be further optimized to be clinically applicable. However, it is likely that this technology will be incorporated in future neurosurgical workflows because the system improves spatial anatomical understanding for the surgeon.

Design and manufacture AR head-mounted displays: A review and outlook

Augmented reality head-mounted displays (AR-HMDs) enable users to see real images of the outside world and visualize virtual information generated by a computer at any time and from any location, making them useful for various applications. The manufacture of AR-HMDs combines the fields of optical engineering, optical materials, optical coating, precision manufacturing, electronic science, computer science, physiology, ergonomics, etc. This paper primarily focuses on the optical engineering of AR-HMDs. Optical combiners and display devices are used to combine real-world and virtual-world objects that are visible to the human eye. In this review, existing AR-HMD optical solutions employed for optical combiners are divided into three categories: optical solutions based on macro-, micro-, and nanooptics. The physical principles, optical structure, performance parameters, and manufacturing process of different types of AR-HMD optical solutions are subsequently analyzed. Moreover, their advantages and disadvantages are investigated and evaluated. In addition, the bottlenecks and future development trends in the case of AR-HMD optical solutions are discussed.

Development and Accuracy Evaluation of Augmented Reality-based Patient Positioning System in Radiotherapy: A Phantom Study.

To develop and evaluate the accuracy of augmented reality (AR)-based patient positioning systems in radiotherapy. AR head-mounted displays (AR-HMDs), which virtually superimpose a three-dimensional (3D) image generated by the digital imaging and communications in medicine (DICOM) data, have been developed. The AR-based positioning feasibility was evaluated. Then, the setup errors of three translational axes directions and rotation angles between the AR and the conventional laser-based positioning were compared. The AR-based pelvic phantom positioning was feasible. The setup errors of AR-based positioning were comparable to laser-based positioning in all translational axis directions and rotation angles. The time necessary for AR-based positioning was significantly longer than that for laser-based positioning (171.0 s vs. 47.5 s, p<0.001). AR-based positioning for radiotherapy was feasible, and showed comparable positioning errors to those of conventional line-based positioning; however, a markedly longer setup time was necessary.

Designing Adaptive Extended Reality Training Systems Based on Expert Instructor Behaviors

Advances in training technologies using Extended Reality (XR) offer dramatic improvements in terms of time and associated costs over traditional training methods. Industries that are reliant on human labor are starting to embrace XR devices such as Virtual and Augmented Reality (VR/AR) Head-Mounted Displays (HMDs) to capitalize on these advantages. Additionally, developers now have the capacity to create customized user experiences and reduce instructor workload by employing adaptive automation in XR training simulations. Adaptive systems analyze the behavior of a learner for a trigger and then deploy an adaptation to alter the system state. While examples of adaptive XR training systems in current literature show that they are feasible to develop, further research must be done to quantify their efficacy. Furthermore, recommendations for how to design triggers and adaptations for XR training simulations are currently non-existent, resulting in negative impacts to training if inappropriate adaptations are implemented. This paper provides novel, evidence-based recommendations for the design of future adaptive XR training systems by learning from expert XR instructors. To this end, semi-structured interviews were conducted with 11 XR trainers. Participants were asked to discuss their experiences dealing with learners who exhibited confusion during XR training. Interviews were analyzed for existing and emerging themes. Finally, these themes were applied to existing trigger and adaptation models and synthesized into design recommendations for XR training developers. The outcomes of this work will inform the future development of adaptive XR training platforms.

Advanced Assistive Maintenance Based on Augmented Reality and 5G Networking.

Internet of Things (IoT) applications play a relevant role in today’s industry in sharing diagnostic data with off-site service teams, as well as in enabling reliable predictive maintenance systems. Several interventions scenarios, however, require the physical presence of a human operator: Augmented Reality (AR), together with a broad-band connection, represents a major opportunity to integrate diagnostic data with real-time in-situ acquisitions. Diagnostic information can be shared with remote specialists that are able to monitor and guide maintenance operations from a control room as if they were in place. Furthermore, integrating heterogeneous sensors with AR visualization displays could largely improve operators’ safety in complex and dangerous industrial plants. In this paper, we present a complete setup for a remote assistive maintenance intervention based on 5G networking and tested at a Vodafone Base Transceiver Station (BTS) within the Vodafone 5G Program. Technicians’ safety was improved by means of a lightweight AR Head-Mounted Display (HDM) equipped with a thermal camera and a depth sensor to foresee possible collisions with hot surfaces and dangerous objects, by leveraging the processing power of remote computing paired with the low latency of 5G connection. Field testing confirmed that the proposed approach can be a viable solution for egocentric environment understanding and enables an immersive integration of the obtained augmented data within the real scene.