Head-mounted Augmented Reality Research Articles

Exploration of the application of augmented reality technology for teaching spinal tumor's anatomy and surgical techniques.

Augmented reality (AR) technology is gradually being applied in surgical teaching as an innovative teaching method. Developing innovative teaching methods to replicate clinical theory and practical teaching scenarios, simulate preoperative planning and training for bone tumor surgery, and offer enhanced training opportunities for young physicians to acquire and apply clinical knowledge is a crucial concern that impacts the advancement of the discipline and the educational standards for young orthopedic physicians. This study explores the application effect of augmented reality technology in anatomy teaching and surgical clinical teaching for spinal tumor. The method utilizes virtual reality and augmented reality technology to present a spinal tumor model and the surgical process of percutaneous vertebroplasty. We conducted a random selection of 12 students forming into the augmented reality teaching group and 13 students forming into the traditional teaching group among the 8-year medical students from Peking Union Medical College and Tsinghua University, ensuring that the age and learning stage of the students in both groups were similar. Two groups of students were taught using traditional teaching methods and augmented reality technology-assisted teaching methods, respectively. A questionnaire survey was conducted after class to assess the quality of course instruction, student motivation in learning, their proficiency in anatomical structures, their comprehension of spinal tumor growth and metastasis, and their understanding and proficiency in percutaneous vertebroplasty. This study was the first to apply augmented reality technology in teaching, using spinal tumors and percutaneous vertebroplasty as examples, a head-mounted augmented reality device was used to create learning scenarios, presenting the complex three-dimensional spatial structure intuitively. The two groups of students differ significantly in their rating of teaching quality, enthusiasm for learning, knowledge of anatomical features, understanding of spinal trabecular structure, and understanding of steps in percutaneous vertebroplasty. The augmented reality technology-assisted teaching system demonstrates outstanding advantages. Augmented reality technology has great potential and broad prospects in teaching bone tumors, which can help improve the visualization, interactivity, and three-dimensional spatial sense of medical teaching in spinal tumor. The application and development prospects of using augmented reality technology for anatomy instruction, surgical teaching, and simulation training are extensive.

Applying augmented reality in teaching of surgical residents-telementoring, a "stress-free" way to surgical autonomy?

Achieving surgical autonomy can be considered the ultimate goal of surgical training. Innovative head-mounted augmented reality (AR) devices enable visualization of the operating field and teaching from remote. Therefore, utilization of AR glasses may be a novel approach to achieve autonomy. The aim of this pilot study is to analyze the feasibility of AR application in surgical training and to assess its impact on intraoperative stress. A head-mounted RealWear Navigator® 500 glasses and the TeamViewer software were used. Initial "dry lab" testing of AR glasses was performed in combination with the Symbionix LAP Mentor™. Subsequently, residents performed various stage-adapted surgical procedures semi-autonomously (SA) (on-demand consultation of senior surgeon, who is in theatre but not scrubbed) versus permanent remote supervision (senior surgeon not present) via augmented reality (AR) glasses, worn by the resident in theatre. Stress was measured by intraoperative heart rate (Polar® pulse belt) and State-Trait Anxiety Inventory (STAI) questionnaire. After "dry lab" testing, N = 5 senior residents performed equally N = 25 procedures SA and with AR glasses. For both, open and laparoscopic procedure AR remote assistance showed satisfactory applicability. Utilization of AR significantly reduced intraoperative peak pulse rate from 131 to 119bpm (p = 0.004), as compared with the semi-autonomous group. Likewise, subjectively perceived stress according to STAI was significantly lower in the AR group (p = 0.011). AR can be applied in surgical training and may help to reduce stress in theatre. In the future, AR has a huge potential to become a stepping stone to surgical autonomy.

Gaze-enabled activity recognition for augmented reality feedback

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material.

Assessment of the Impact of a Head-mounted Augmented Reality Low Vision Aid on Vision and Quality of Life in Children and Young People with Visual Impairment.

Electronic head-mounted low vision aids (LVAs) can help children and young people (CYP) to access schoolwork and leisure activities which they would otherwise struggle to be able to do with traditional optical or hand held LVAs. SightPlus uses a smartphone mounted in a virtual reality headset controlled using a Bluetooth joystick. It offers users 0.7-24.3× magnification alongside enhanced modes to maximise vision. Eighteen participants aged 8-16 years with reduced vision were given SightPlus to use at home for four weeks. Visual acuity was assessed with and without SightPlus along with reading performance, contrast sensitivity, functional vision and quality of life questionnaires. Clinically significant improvements in distance vision (0.633logMAR SD ± 0.359), near vision (0.411logMAR SD ± 0.368), reading acuity (0.454LlogMAR SD ± 0.406) and critical print size (0.285logMAR ± 0.360) were seen when testing with SightPlus.However, there was a mean decrease in contrast sensitivity and reading speed when using SightPlus. Despite this, nine out of the 14 patients included for analysis indicated a preference to continue to use SightPlus. Of note, younger participants were more likely to show a preference for using SightPlus. All seven CYP aged 10 or under wanted to continue to use SightPlus; in contrast, only two of the seven participants aged 11 or over wanted to continue. Like the results in adult populations, SightPlus has been found to improve CYP visual functions. Older participants were less likely to want to continue to use SightPlus, potentially suggesting they have found other methods for managing sight loss.

Enhancing Security and Privacy in Head-Mounted Augmented Reality Systems Using Eye Gaze

Augmented Reality (AR) devices offer a rich, immersive experience that provides the user with a blend of the physical and the synthetic, digitally augmented world. This augmentation is made possible by the AR device's sensors, which are constantly absorbing information about both the user's physical environment (using sensors such as depth and photographic cameras) and about the users themselves (using eye gaze sensors, for example). This information presents both problems and unique solutions. For example, user eye gaze has been shown to be effective in creating efficient and usable local pairing methods between two or more devices and to provide salience in determining what visual data should not be included in raw sensor outputs.

Development of an augmented reality guidance system for head and neck cancer resection.

The use of head-mounted augmented reality (AR) for surgeries has grown rapidly in recent years. AR aids in intraoperative surgical navigation through overlaying three-dimensional (3D) holographic reconstructions of medical data. However, performing AR surgeries on complex areas such as the head and neck region poses challenges in terms of accuracy and speed. This study explores the feasibility of an AR guidance system for resections of positive tumour margins in a cadaveric specimen. The authors present an intraoperative solution that enables surgeons to upload and visualize holographic reconstructions of resected cadaver tissues. The solution involves using a 3D scanner to capture detailed scans of the resected tissue, which are subsequently uploaded into our software. The software converts the scans of resected tissues into specimen holograms that are viewable through a head-mounted AR display. By re-aligning these holograms with cadavers with gestures or voice commands, surgeons can navigate the head and neck tumour site. This workflow can run concurrently with frozen section analysis. On average, the authors achieve an uploading time of 2.98min, visualization time of 1.05min, and re-alignment time of 4.39min, compared to the 20 to 30min typical for frozen section analysis. The authors achieve a mean re-alignment error of 3.1mm. The authors' software provides a foundation for new research and product development for using AR to navigate complex 3D anatomy in surgery.

Invited Session IV: Extended reality--applications in vision science and beyond: Augmented reality systems for people with low vision.

Low vision is a visual impairment that falls short of blindness but cannot be corrected by eyeglasses or contact lenses. While current low vision aids (e.g., magnifier, CCTV) support basic vision enhancements, such as magnification and contrast enhancement, these enhancements often arbitrarily alter a user's full field of view without considering the user's context, such as their visual abilities, tasks, and environmental factors. As a result, these low vision aids are not sufficient or preferred by low vision users in many important tasks. Augmented reality (AR) technology presents a unique opportunity to enhance low vision people's visual experience by automatically recognizing the surrounding environment and presenting tailored visual augmentations. In this talk, I will talk about how we design and build intelligent AR systems to support low vision people in visual tasks, such as a head-mounted AR system that presents visual cues to orient users' attention in a visual search task, as well as a projection-based AR system that projects visual highlights on the stair edges to support safe stair navigation. I will conclude my talk by discussing our future research direction on AR for low vision accessibility.

Development of the Troop Readiness Evaluation With Augmented Reality Return-to-Duty (Troop READY) Platform to Aid in the Detection and Treatment of Military Mild Traumatic Brain Injury.

Mild traumatic brain injury (mTBI) is prevalent in service members (SMs); however, there is a lack of consensus on the appropriate approach to return to duty (RTD). Head-mounted augmented reality technology, such as the HoloLens 2, can create immersive, salient environments to more effectively evaluate relevant military task performance. The Troop Readiness Evaluation with Augmented Reality Return-to-Duty (READY) platform was developed to objectively quantify cognitive and motor performance during military-specific activities to create a comprehensive approach to aid in mTBI detection and facilitate appropriate RTD. The aim of this project was to detail the technical development of the Troop READY platform, the outcomes, and its potential role in the aiding detection and RTD decision-making post mTBI. The secondary aim included evaluating the safety, feasibility, and SM usability of the Troop READY platform. The Troop READY platform comprises three assessment modules of progressing complexity: (1) Static and Dynamic Mobility, (2) Rifle Qualification Test, and (3) Small Unit Operations Capacity-Room Breach/Clearing Exercise. The modules were completed by 137 active duty SMs. Safety was assessed through monitoring of adverse events. Feasibility was assessed using the self-directed module completion rate. Usability was measured using the Systems Usability Scale. No adverse events occurred. Completion rates of the three modules ranged from 98 to 100%. In terms of usability, the mean Systems Usability Scale score of all participants was 83.92 (13.95), placing the Troop READY platform in the good-to-excellent category. Objective motor and cognitive outcomes were generated for each module. The Troop READY platform delivers self-directed, salient assessment modules to quantify single-task, dual-task, and unit-based performance in SMs. The resultant data provide insight into SM performance through objective outcomes and identify specific areas of executive or motor function that may be slow to recover following mTBI.

Augmented reality, deep learning and vision-language query system for construction worker safety

Low situational awareness contributes to safety incidents in construction. Existing Deep Learning (DL)-based applications lack the capability to provide context-specific and interactive feedback that is essential for workers to fully understand their surrounding environments. This paper proposes the Visual Construction Safety Query (VCSQ) system. The system encompasses real-time Image Captioning (IC), safety-centric Visual Question Answering (VQA), and keyword-based Image-Text Retrieval (ITR), integrated with head-mounted Augmented Reality (AR) devices. System validation includes benchmarks and real-world images. The ITR module posted high recall rates of 0.801 and 0.835 for Recall@5 and @10. The VQA module achieved an 89.7% accuracy rate, and the IC module had a SPICE score of 0.449. Feasibility tests and surveys confirmed the system's practical advantages in different construction scenarios. This study establishes an integration roadmap adaptable to future advancements in interactive DL and immersive AR.

A First Exploration on the Use of Head-Mounted Augmented Reality in the Context of the Portuguese Military

In this paper, we present the design and implementation of a first iteration of an augmented reality (AR) system for dismounted soldiers in the Portuguese military. We started the work via a survey of 86 members of the military to better understand their experience, needs, and preferences with current Command & Control (C2) systems. We then assessed the effects of our prototype on the performance, situational awareness, and perceived usability and workload of 13 participants from a local Commando Regiment. We compared our results to a representative baseline using a paper map and radio in a hostage extraction simulation and found that our first AR iteration, despite a short practice session, increased the quality of the information available and decreased the complexity, temporal demands, and effort required to complete the study tasks; leading to an overall decrease in perceived workload. Overall, participants described the AR experience as more user-friendly. We conclude our case study with research ideas for further iterations of our prototype.

Augmented reality navigation-guided intraoperative pulmonary nodule localization: a pilot study.

With the increasing number of small pulmonary nodules detected, effective localization of pulmonary nodules has become an issue. The goal of this study is to determine the safety and feasibility of a newly developed augmented reality navigation technology for intraoperative localization of small pulmonary nodules. We conducted a prospective single-center feasibility study of a novel augmented reality navigation system and lung localization (LungBrella) marker on ten patients between July and October 2020. For augmented reality navigation-guided localization, a preoperative chest computed tomography scan was performed to generate 3-dimensional (3D) virtual images and individualized localization plan, which were uploaded into Hololens (a head-mounted augmented reality device). Under the guidance of established procedure plan displayed by HoloLens, localization marker was placed in operating room. Segmentectomy or wedge resection was subsequently performed. The primary endpoint was the localization procedure success rate, and the secondary endpoints were localization time, operation time, and complications. Localization was successful in seven of the ten procedures. Due to different reasons, failures were noted in three cases, after which immediate adjustments were made. In the successful cases, the LungBrella marker was positioned at a median of 5.8 mm (range, 0-10 mm) from the edge of the nodule. Median localization time was 9.4 min (range, 5-19 min), and median operation time was 172.9 min (range, 105-200 min). There were no complications during the entire process. This exploratory study suggests that augmented reality navigation-guided pulmonary nodule localization is a safe and feasible technique (ClinicalTrials.gov identifier, NCT04211051).

Head-mounted augmented reality to support reassurance and social interaction for autistic children with severe learning disabilities

Augmented Reality (AR) is promising to complement autism approaches, but so far has mainly focused on training socio-emotional abilities for autistic children with mild learning disabilities. To better consider autistic children with severe learning disabilities and complex needs (SLN), stakeholders advise using collaborative AR sensory-based mediation approaches. Magic Bubbles is a multisensory AR environment created based on stakeholders’ interviews, then adapted for a day hospital setting in collaboration with practitioners, and finally validated in terms of acceptability and usability for autistic children with SLN. In this paper, we report on our latest study that explores three main research questions: 1) To what extent can Magic Bubbles secure autistic children with SLN? 2) To what extent can Magic Bubbles prompt the dyadic relationship between an autistic child with SLN and a practitioner? 3) What is the overall quality of experience for autistic children with SLN when using Magic Bubbles? To answer these questions, seven autistic children with SLN participated in at least six weekly sessions over three months in a day hospital setting. Data collection and analysis used qualitative and quantitative methods, mainly drawing upon grounded theory to evaluate their experiences. Findings validate the three research questions, offer a detailed account of children’s experiences with AR, and outline future directions.

Hand-Controlled User Interfacing for Head-Mounted Augmented Reality Learning Environments

With the rapid expansion of technology and hardware availability within the field of Augmented Reality, building and deploying Augmented Reality learning environments has become more logistically viable than ever before. In this paper, we focus on the development of a new mobile learning experience for a museum by combining multiple technologies to provide additional Human–computer interaction possibilities. This is both to reduce barriers to entry for end-users as well as provide natural interaction methods. Using our method, we implemented a new approach to gesture-based interactions for Augmented Reality interactions by combining two devices, a Leap Motion and a Microsoft HoloLens (1st Generation), via an intermediary device with the use of local-area networking. This was carried out with the intention of comparing this method against alternative forms of Augmented Reality to determine which implementation has the largest impact on adult learners’ ability to retain information. A control group has been used to establish data on memory retention without the use of Augmented Reality technology, along with three focus groups to explore the different methods and locations. Results found that adult learners retain the most overall information when being educated through a traditional lecture, with a statistically significant difference between the methods; however, the use of Augmented Reality resulted in a slower rate of knowledge decay between testing intervals. This contrasts with existing research as adult learners did not respond to the technology in the same way that child and teenage audiences previously have, which suggests that prior research may not be generalisable to all audiences.

Evaluating Augmented Reality Landmark Cues and Frame of Reference Displays with Virtual Reality.

Daily travel usually demands navigation on foot across a variety of different application domains, including tasks like search and rescue or commuting. Head-mounted augmented reality (AR) displays provide a preview of future navigation systems on foot, but designing them is still an open problem. In this paper, we look at two choices that such AR systems can make for navigation: 1) whether to denote landmarks with AR cues and 2) how to convey navigation instructions. Specifically, instructions can be given via a head-referenced display (screen-fixed frame of reference) or by giving directions fixed to global positions in the world (world-fixed frame of reference). Given limitations with the tracking stability, field of view, and brightness of most currently available head-mounted AR displays for lengthy routes outdoors, we decided to simulate these conditions in virtual reality. In the current study, participants navigated an urban virtual environment and their spatial knowledge acquisition was assessed. We experimented with whether or not landmarks in the environment were cued, as well as how navigation instructions were displayed (i.e., via screen-fixed or world-fixed directions). We found that the world-fixed frame of reference resulted in better spatial learning when there were no landmarks cued; adding AR landmark cues marginally improved spatial learning in the screen-fixed condition. These benefits in learning were also correlated with participants' reported sense of direction. Our findings have implications for the design of future cognition-driven navigation systems.

559 Novel Technique of Head-Mounted Augmented Reality Assisted Cerebral Angiogram: Proof of Concept on a Flow Model

INTRODUCTION: Augmented reality (AR) is an emerging technology in medicine while still under-explored in the endovascular field. METHODS: A 5 French guide was delivered to the proximal ICA-equivalent vessel. DynaCT was performed for the flow model using a uniplanar Artis Q angiogram suite (Siemens Medical Solutions, Erlangen, Germany) to create a CTA to act as a surrogate for non-invasive CTA in patients. A 3D reconstruction of the circulation was performed using the Syngo station (Siemens Medical Solutions, Erlangen, Germany). After importing the 3D image to the HMAR system, a hologram of the circulation was created. A roadmap image was created and the AR hologram was superimposed to it on the angiography screen. A 0.012” microcatheter with a microwire was then advanced to the tip of the guide catheter. RESULTS: The operator navigated the proximal and distal intracranial vessels using the hologram to replace the roadmap; and fluoroscopy guidance for visualization of the catheters and microwire. No contrast was needed as the hologram acted as the roadmap. The microcatheter was navigated successfully to distal targets in the ipsilateral and contralateral intracranial vasculature without the need to repeat road maps or inject contrast. CONCLUSIONS: The concept of HMAR-assisted cerebral angiography is feasible. We were able to perform the whole intracranial navigation using one pre-operative CTA only. This technology needs further development and has huge potential to decrease contrast use and possibly radiation exposure.

Effect of visual field asymmetries on performance while utilizing aircraft attitude symbology

This research explores the effect of symbology placement on human performance for users of head-mounted augmented reality displays. A pair of experiments examined the impact on visual performance asymmetries when perceiving complex, meaningful visual stimuli, such as the Arc Segment Attitude Reference (ASAR). The ASAR symbology represents an aircraft’s vertical flight path and roll angles. Experiment 1 examined participants’ performance in making categorical and coordinate judgments regarding various attitudes of the ASAR and a Gabor patch which were briefly presented in the peripheral visual field. The results were consistent with the horizontal-vertical anisotropy literature, which implies that performance would be better for stimuli placed on the horizontal than the vertical meridian. Experiment 2 assessed asymmetries for continuously presented stimuli in a dual-task environment which involved a centrally located, demanding visual psychomotor task and monitoring of the ASAR or Gabor stimuli at the same peripheral locations as Experiment 1. No performance differences were found as a function of peripheral stimulus placement. However, eye tracking, particularly for a subset of the participants suggest they employed a more efficient visual process to monitor the peripheral stimuli when the stimuli were placed on the horizontal meridian.

Prototyping to elicit user requirements for product development: Using head-mounted augmented reality when designing interactive devices

Designers of interactive devices are challenged by the need to accurately elicit user requirements from low-cost prototypes at the early stages of the design process. Head-mounted augmented reality (AR) can potentially assist in this process by economically representing physical-digital blended features with relatively high-fidelity prototypes. To explore this potential, we present and evaluate a head-mounted AR-enhanced hybrid prototyping system created in the context of a fan product development process. We conducted a mixed-methods study comparing the AR-enhanced prototyping method with a conventional prototyping method. The results reveal that the AR system can elicit similar user requirements as the conventional prototyping method with an improved overall experience.

Head-Mounted Augmented Reality Support for Assemblers of Wooden Trusses

Wooden-house assembly is an area where still a big part of the work is done manually. In this case study, pairs of operators compose large wooden pieces together based on paper-print instructions complemented by visual guidance in the shape of laser marks projected from lasers mounted in the ceiling, based on Computer-aided design (CAD) data. Augmented Reality (AR) head-mounted displays (HMD) offer a unique platform for providing instructions and additional information superimposed in the work environment and thus can provide guidance in a cognitively ergonomic way. A particular advantage compared to other computing platforms is that the operators have free hands and can perform the manual work and follow guidance simultaneously. We present an evaluation of a prototype that dynamically transforms a CAD data file with design and measurements of wooden trusses to be manufactured, into an AR-based guidance system developed in Unity for Microsoft HoloLens 2 devices. We used an iterative participatory design process for prototyping and think-aloud protocol combined with observations for evaluation, involving professional assemblers in different stages of the process. Participants found the solution to potentially save time in their everyday work and simplify the task by offering increased visibility of the marks compared to the existing laser projection. Large-scale deployment of the system is still facing design challenges of which some are also discussed in the paper.

Augmented Reality for AI-driven Inspection? – A Comparative Usability Study

Inspection in Aerospace industry can, as well as many other industrial applications, benefit from using Augmented Reality (AR) due to its ability to superimpose helpful digital information in 3D, leading to fewer errors and decreased mental demand. However, each AR device has advantages and disadvantages, and not all AR devices are suitable for use in industrial settings. We compare a tripod-fitted-adjustable-arm tablet-based AR solution (Apple iPad Pro) to head-mounted AR (Microsoft HoloLens 2) and a traditional, computer screen-based human-machine interface (HMI), all three designed to guide operators based on previously performed AI-based image analysis. Following an iterative design process with three formative evaluations, a final field test in a real industrial shop floor engaging 6 professional inspectors revealed an overall preference for the tripod-fitted iPad variant which receiving the best scores in most dimensions covered in both a usability-focused SUS questionnaire (score 71) and a NASA-RTLX form focused on perceived workload. More specifically, the tripod-fitted iPad was considered more usable (SUS) than the classic computer display HMI (M=5.83, SD=4.92, p=0.034, N=6); the temporal demand (NASA-RTLX) was considered lower using the iPad compared to both HoloLens 2 and the HMI (M=6.67, SD=4.08, p=0.010; M=10.83, SD=9.70, p=0.040, N=6), respectively.

RETRACTED: Head-Mounted Augmented Reality in the Planning of Cerebrovascular Neurosurgical Procedures: A Single-Center Initial Experience

Augmented reality (AR) technology has played an increasing role in cerebrovascular neurosurgery over the last 2decades. Hence, we aim to evaluate the technical and educational value of head-mounted AR in cerebrovascular procedures. This is a single-center retrospective study of patients who underwent open surgery for cranial and spinal cerebrovascular lesions between April and August 2022. In all cases, the Medivis Surgical AR platform and HoloLens 2 were used for preoperative and intraoperative (preincision) planning. Surgical plan adjustment due to the use of head-mounted AR and subjective educational value of the tool were recorded. A total of 33 patients and 35 cerebrovascular neurosurgical procedures were analyzed. Procedures included 12 intracranial aneurysm clippings, 6 brain and 1 spinal arteriovenous malformation resections, 2 cranial dural arteriovenous fistula obliterations, 3 carotid endarterectomies, two extracranial-intracranial direct bypasses, two encephaloduroangiosynostosis for Moyamoya disease, 1 biopsy of the superficial temporal artery, 2 microvascular decompressions, 2 cavernoma resections, 1 combined intracranial aneurysm clipping and encephaloduroangiosynostosis for Moyamoya disease, and 1 percutaneous feeder catheterization for arteriovenous malformation embolization. Minor changes in the surgical plan were recorded in 16 of 35 procedures (45.7%). Subjective educational value was scored as "very helpful" for cranial, spinal arteriovenous malformations, and carotid endarterectomies; "helpful" for intracranial aneurysm, dural arteriovenous fistulas, direct bypass, encephaloduroangiosynostosis, and superficial temporal artery-biopsy; and "not helpful" for cavernoma resection and microvascular decompression. Head-mounted AR can be used in cerebrovascular neurosurgery as an adjunctive tool that might influence surgical strategy, enable 3-dimensional understanding of complex anatomy, and provide great educational value in selected cases.