Augmented Reality Head-mounted Display Research Articles

Performance Analysis of Light Detection and Ranging (LIDAR) in Augmented Reality (AR)

Augmented reality (AR) has existed for decades, dating back from roughly 50 years ago when the renowned ‘father of computer graphics’ Ivan Sutherland first invented an AR head-mounted display system. Ever since, various organizations strive to improve the technology but while the AR technology definitely evolved significantly since its advent, the technology progress is relatively stagnant these past few years whereby we see little to no usage at least from common end user perspective. This may be due to various limitations of technologies currently utilized in AR. However, things may change since in 2020, the tech giant, Apple Inc. has made an unprecedented breakthrough by including the light detection and ranging (LiDAR) scanner on consumers’ smartphones which could potentially revolutionize the AR industry. This research primarily revolves around analyzing the performance of utilization of LiDAR technology in AR mainly in mobile devices available for the mass. The methodology and concept for this research is fairly straight-forward, the AR experience boosted by the newly introduced LiDAR technology will be compared with the widely used technology which primarily consist of the implementation of Simultaneous Localization and Mapping (SLAM) and RGB camera with the exclusion of LiDAR and assessing the potential of LiDAR to further improvise AR experience and implementation in various fields. Testing results has shown significant improvement from various important aspects in AR whereby occlusion is possible in object rendering, the estimations based on live visual stimuli accuracy at roughly 99% in comparison to the true values as well as the ability to accurately perceive the geometry of objects in the real world environment through advanced depth and space perception.

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays.

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

A usability analysis of augmented reality and haptics for surgical planning.

Proper visualization and interaction with complex anatomical data can improve understanding, allowing for more intuitive surgical planning. The goal of our work was to study what the most intuitive yet practical platforms for interacting with 3D medical data are in the context of surgical planning. We compared planning using a monitor and mouse, a monitor with a haptic device, and an augmented reality (AR) head-mounted display which uses a gesture-based interaction. To determine the most intuitive system, two user studies, one with novices and one with experts, were conducted. The studies involved planning of three scenarios: (1) heart valve repair, (2) hip tumor resection, and (3) pedicle screw placement. Task completion time, NASA Task Load Index and system-specific questionnaires were used for the evaluation. Both novices and experts preferred the AR system for pedicle screw placement. Novices preferred the haptic system for hip tumor planning, while experts preferred the mouse and keyboard. In the case of heart valve planning, novices preferred the AR system but there was no clear preference for experts. Both groups reported that AR provides the best spatial depth perception. The results of the user studies suggest that different surgical cases may benefit from varying interaction and visualization methods. For example, for planning surgeries with implants and instrumentations, mixed reality could provide better 3D spatial perception, whereas using landmarks for delineating specific targets may be more effective using a traditional 2D interface.

Multi-Dimensional Evaluation of an Augmented Reality Head-Mounted Display User Interface for Controlling Legged Manipulators

Controlling assistive robots can be challenging for some users, especially those lacking relevant experience. Augmented Reality (AR) User Interfaces (UIs) have the potential to facilitate this task. Although extensive research regarding legged manipulators exists, comparatively little is on their UIs. Most existing UIs leverage traditional control interfaces such as joysticks, Hand-Held (HH) controllers and 2D UIs. These interfaces not only risk being unintuitive, thus discouraging interaction with the robot partner, but also draw the operator’s focus away from the task and towards the UI. This shift in attention raises additional safety concerns, particularly in potentially hazardous environments where legged manipulators are frequently deployed. Moreover, traditional interfaces limit the operators’ availability to use their hands for other tasks. Towards overcoming these limitations, in this article, we provide a user study comparing an AR Head-Mounted Display (HMD) UI we developed for controlling a legged manipulator against off-the-shelf control methods for such robots. This user study involved 27 participants and 135 trials, from which we gathered over 405 completed questionnaires. These trials involved multiple navigation and manipulation tasks with varying difficulty levels using a Boston Dynamics’s Spot, a 7 df Kinova robot arm and a Robotiq 2F-85 gripper that we integrated into a legged manipulator. We made the comparison between UIs across multiple dimensions relevant to a successful human–robot interaction. These dimensions include cognitive workload, technology acceptance, fluency, system usability, immersion and trust. Our study employed a factorial experimental design with participants undergoing five different conditions, generating longitudinal data. Due to potential unknown distributions and outliers in such data, using parametric methods for its analysis is questionable, and while non-parametric alternatives exist, they may lead to reduced statistical power. Therefore, to analyse the data that resulted from our experiment, we chose Bayesian data analysis as an effective alternative to address these limitations. Our results show that AR UIs can outpace HH-based control methods and reduce the cognitive requirements when designers include hands-free interactions and cognitive offloading principles into the UI. Furthermore, the use of the AR UI together with our cognitive offloading feature resulted in higher usability scores and significantly higher fluency and Technology Acceptance Model scores. Regarding immersion, our results revealed that the response values for the AR Immersion questionnaire associated with the AR UI are significantly higher than those associated with the HH UI, regardless of the main interaction method with the former, i.e., hand gestures or cognitive offloading. Derived from the participants’ qualitative answers, we believe this is due to a combination of factors, of which the most important is the free use of the hands when using the HMD, as well as the ability to see the real environment without the need to divert their attention to the UI. Regarding trust, our findings did not display discernible differences in reported trust scores across UI options. However, during the manipulation phase of our user study, where participants were given the choice to select their preferred UI, they consistently reported higher levels of trust compared to the navigation category. Moreover, there was a drastic change in the percentage of participants that selected the AR UI for completing this manipulation stage after incorporating the cognitive offloading feature. Thus, trust seems to have mediated the use and non-use of the UIs in a dimension different from the ones considered in our study, i.e., delegation and reliance. Therefore, our AR HMD UI for the control of legged manipulators was found to improve human–robot interaction across several relevant dimensions, underscoring the critical role of UI design in the effective and trustworthy utilisation of robotic systems.

Requirement analysis for an AI-based AR assistance system for surgical tools in the operating room: stakeholder requirements and technical perspectives

PurposeWe aim to investigate the integration of augmented reality (AR) within the context of increasingly complex surgical procedures and instrument handling toward the transition to smart operating rooms (OR). In contrast to cumbersome paper-based surgical instrument manuals still used in the OR, we wish to provide surgical staff with an AR head-mounted display that provides in-situ visualization and guidance throughout the assembly process of surgical instruments. Our requirement analysis supports the development and provides guidelines for its transfer into surgical practice.MethodsA three-phase user-centered design approach was applied with online interviews, an observational study, and a workshop with two focus groups with scrub nurses, circulating nurses, surgeons, manufacturers, clinic IT staff, and members of the sterilization department. The requirement analysis was based on key criteria for usability. The data were analyzed via structured content analysis.ResultsWe identified twelve main problems with the current use of paper manuals. Major issues included sterile users’ inability to directly handle non-sterile manuals, missing details, and excessive text information, potentially delaying procedure performance. Major requirements for AR-driven guidance fall into the categories of design, practicability, control, and integration into the current workflow. Additionally, further recommendations for technical development could be obtained.ConclusionIn conclusion, our insights have outlined a comprehensive spectrum of requirements that are essential for the successful implementation of an AI- and AR-driven guidance for assembling surgical instruments. The consistently appreciative evaluation by stakeholders underscores the profound potential of AR and AI technology as valuable assistance and guidance.

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

Forming a view: a human factors case study of augmented reality collaboration in assembly

Industry 4.0 technology is promoted as improving manufacturing flexibility, and competitiveness; though Australia has been slow to adopt. The Australian Navy shipbuilding program provides opportunities for accelerating technology adoption, revitalising manufacturing productivity and competitiveness. Adopting a sociotechnical systems lens, our research sought to identify usability, workload, and user experience of an augmented reality head-mounted display (AR-HMD) deployed to complete multiple work tasks in a workflow (electrical assembly, collaborative robot (cobot) mediated inspection, and remote troubleshooting using video call). Usability was rated ‘average’ (System Usability Scale mean = 69.8) and workload ‘acceptable’ (NASA Task Load Index mean = 25.8) for the AR-HMD alone, with usability of the integrated work system (IWS) rated ‘good’ (SUS mean = 79.2). Results suggest software interfaces, tracking, and gesturing methods for the AR-HMD require improvement. This trial shows the AR-HMD provides a versatile platform for integrating multiple digital technologies without hindering effectiveness of end-user performance, potentially benefiting productivity and quality.

Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance.

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

Compensated DOE in a VHG-based waveguide display to improve uniformity

Augmented reality head-mounted displays (AR-HMDs) utilizing diffractive waveguides have emerged as a popular research focus. However, the illuminance uniformity over the fields of view (FOV) is often unsatisfactory in volume holographic grating (VHG) based waveguide displays. This paper proposes a high uniformity AR waveguide display system. Firstly, the angular uniformity of the VHG-based waveguide displays is analyzed. Subsequently, diffractive optical elements (DOEs) are seamlessly integrated onto the outer coupling surface of the waveguide substrate to improve the angular uniformity through phase compensation. To design the DOE phase, the multi-objective stochastic gradient descent (MO-SGD) algorithm is proposed. A single DOE is used to compensating various images form the image source. A hybrid loss, which includes the learned perceptual image patch similarity (LPIPS) metric, is applied to enhance the algorithm performance. Simulation results show that the proposed method effectively suppresses illumination degradation at the edge FOV in exit pupil images of the waveguide display system. In the results, the peak signal-to-noise ratio (PSNR) is improved by 5.54 dB. Optical experiments validate the effectiveness of the proposed method. The measured nonuniformity (NU) against FOVs is improved by 53.05% from 0.3749 to 0.1760.

1071 Initial Experience Using an Augmented Reality Head-Mounted Display System During Surgical Management of Thoracolumbar Spinal Trauma

INTRODUCTION: Advancements in trauma medicine can improve the speed and accuracy of care response with a major impact on downstream operative outcomes. Augmented reality using a head-mounted-display (AR-HMD) is a novel technology that may be applied to spine surgery to provide surgeons with a real-time CT-guided 3-dimensional reconstruction of a patient’s spinal anatomy. METHODS: We retrospectively reviewed trauma patients who underwent pedicle screw placement guided by AR-HMD or robotic-assisted platforms at an academic tertiary care center between January 1, 2021 and December 31, 2022. Demographic, clinical, cost, and perioperative outcome information was analyzed and outcome profiles were compared using a U test. RESULTS: The AR cohort (4 females, 5 males) had a mean age of 66.0 years, BMI of 29.4 kg/m2, Charlson Comorbidity Index (CCI) of 4.11, and Surgical Invasiveness Index (SII) of 8.78. Pedicle screws (77) were placed in the thoracic (51%), lumbar (39%), and sacral spine (10%). Intraoperatively, there was a mean blood loss of 377.78 ml, 0.78 units transfused, 398.33 minutes spent in the operating room, and a 20.00 day LOS. The average 3-month, 6-month, and 12-month ODI scores were 14.25 (n = 8), 15.40 (n = 5), and 12.66 (n = 3). The robotic cohort (5 females, 8 males) had a mean age of 55.6 years, BMI of 27.1 kg/m2, CCI of 3.77, and SII of 14.23. Pedicle screws (128) were placed in the thoracic (59%), lumbar (37%), sacral spine (2%), iliac spine (2%), and pelvis (2%). Intra-operatively, there was a mean blood loss of 431.54 ml, 0.46 units transfused, 330.61 minutes spent in the operating room, and a 10.38 day LOS. The average 3-month, 6-month, and 12-month ODI scores were 22.67 (n = 9), 18.60 (n = 5), and 29.50 (n = 2). CONCLUSIONS: This case series demonstrates the utility of AR-HMD in surgical spinal trauma cases by showing operative outcomes comparable to those observed in robotic-assisted surgery.

Clinical Application of an Augmented Reality Navigation System for Transforaminal Epidural Injection: A Randomized Controlled Trial.

Objectives: Augmented reality (AR) navigation systems are emerging to simplify and enhance the precision of medical procedures. Lumbosacral transforaminal epidural injection is a commonly performed procedure for the treatment and diagnosis of radiculopathy. Accurate needle placement while avoiding critical structures remains a challenge. For this purpose, we conducted a randomized controlled trial for our augmented reality navigation system. Methods: This randomized controlled study involved 28 patients, split between a traditional C-arm guided group (control) and an AR navigation guided group (AR-NAVI), to compare procedure efficiency and radiation exposure. The AR-NAVI group used a real-time tracking system displaying spinal structure and needle position on an AR head-mounted display. The procedural time and C-arm usage (radiation exposure) were measured. Results: All patients underwent successful procedures without complications. The AR-NAVI group demonstrated significantly reduced times and C-arm usage for needle entry to the target point (58.57 ± 33.31 vs. 124.91 ± 41.14, p < 0.001 and 3.79 ± 1.97 vs. 8.86 ± 3.94, p < 0.001). Conclusions: The use of the AR navigation system significantly improved procedure efficiency and safety by reducing time and radiation exposure, suggesting a promising direction for future enhancements and validation.

Utilization of Augmented Reality Head-Mounted Display for the Surgical Management of Thoracolumbar Spinal Trauma.

Background and Objectives: Augmented reality head-mounted display (AR-HMD) is a novel technology that provides surgeons with a real-time CT-guided 3-dimensional recapitulation of a patient's spinal anatomy. In this case series, we explore the use of AR-HMD alongside more traditional robotic assistance in surgical spine trauma cases to determine their effect on operative costs and perioperative outcomes. Materials and Methods: We retrospectively reviewed trauma patients who underwent pedicle screw placement surgery guided by AR-HMD or robotic-assisted platforms at an academic tertiary care center between 1 January 2021 and 31 December 2022. Outcome distributions were compared using the Mann-Whitney U test. Results: The AR cohort (n = 9) had a mean age of 66 years, BMI of 29.4 kg/m2, Charlson Comorbidity Index (CCI) of 4.1, and Surgical Invasiveness Index (SII) of 8.8. In total, 77 pedicle screws were placed in this cohort. Intra-operatively, there was a mean blood loss of 378 mL, 0.78 units transfused, 398 min spent in the operating room, and a 20-day LOS. The robotic cohort (n = 13) had a mean age of 56 years, BMI of 27.1 kg/m2, CCI of 3.8, and SII of 14.2. In total, 128 pedicle screws were placed in this cohort. Intra-operatively, there was a mean blood loss of 432 mL, 0.46 units transfused units used, 331 min spent in the operating room, and a 10.4-day LOS. No significant difference was found between the two cohorts in any outcome metrics. Conclusions: Although the need to address urgent spinal conditions poses a significant challenge to the implementation of innovative technologies in spine surgery, this study represents an initial effort to show that AR-HMD can yield comparable outcomes to traditional robotic surgical techniques. Moreover, it highlights the potential for AR-HMD to be readily integrated into Level 1 trauma centers without requiring extensive modifications or adjustments.

Fast, Accurate, But Sometimes Too-Compelling Support: The Impact of Imperfectly Automated Cues in an Augmented-Reality Head-Mounted Display on Visual Search Performance

Fast, Accurate, But Sometimes Too-Compelling Support: The Impact of Imperfectly Automated Cues in an Augmented-Reality Head-Mounted Display on Visual Search Performance

Information Access Effort: The Role of Head Movements for Information Presented at Increasing Eccentricity on Flat Panel and Head-Mounted Displays.

This experiment examined performance costs when processing two sources of information positioned at increasing distances using a flat panel display and an augmented reality head-mounted display (AR-HMD). The AR-HMD enables positioning virtual information at various distances in space. However, the proximity compatibility principle suggests that closer separation when two sources of information require mental integration assists performance, whereas increased separation between two sources hurts integration performance more than when a single source requires focused attention. Previous studies have provided inconsistent findings regarding costs associated with increased separation. Few of these experiments have examined separation for both focused and integration tasks, compared vertical and lateral separation, or measured head movements. Three experiments collectively examined these issues using a flat panel display and a virtual display presented with an HMD, where the separation of information varied laterally or vertically during a focused attention (digit reading) task and an information integration (mental subtraction) task. There was no performance cost for either display when information was increasingly separated. However, head movements mitigated performance costs by preserving accuracy at larger separations without increasing response time. Head movements appear to mitigate performance costs associated with presenting information increasingly far apart on flat panel displays and HMDs. Both eye scanning and head movements appear to be less effortful than expected. These findings have important implications for design guidelines regarding the placement of information presented on flat panel displays and, more specifically, HMDs, which can present information 360° around the user.

Objective Image Quality Optimization in Augmented Reality Using Spatial Frequency Domain Models.

Augmented reality (AR) blends the digital and physical worlds by overlapping a virtual image onto the see-through physical environment. However, contrast reduction and noise superposition in an AR head-mounted display (HMD) can substantially limit image quality and human perceptual performance in both the digital and physical spaces. To assess image quality in AR, we performed human and model observer studies for various imaging tasks with targets placed in the digital and physical worlds. A target detection model was developed for the complete AR system including the optical see-through. Target detection performance using different observer models developed in the spatial frequency domain was compared with the human observer results. The non-prewhitening model with eye filter and internal noise results closely track human perception performance as measured by the area under the receiver operating characteristic curve (AUC), especially for tasks with high image noise. The AR HMD non-uniformity limits the low-contrast target (less than 0.02) observer performance for low image noise. In augmented reality conditions, the detectability of a target in the physical world is reduced due to the contrast reduction by the overlaid AR display image (AUC less than 0.87 for all the contrast levels evaluated). We propose an image quality optimization scheme to optimize the AR display configurations to match observer detection performance for targets in both the digital and physical worlds. The image quality optimization procedure is validated using both simulation and bench measurements of a chest radiography image with digital and physical targets for various imaging configurations.

Augmented Reality Order Picking Aid for Foreign Workers in Warehouses

Background: In warehouse logistics, picking work comprises roughly 60% of the costs, emphasizing the need for efficiency (Matsumoto et al., 2019). Traditionally, workers utilize written instructions, which can be challenging for those less language proficient. The integration of augmented reality (AR) head- mounted displays (HMDs) may enhance accuracy and efficiency (Matsumoto et al., 2019). However, AR HMDs may come with some drawbacks—after extended use, some users report discomfort and decreased performance (Vidovič &amp; Gajšek, 2020). Despite promising improvements in warehouse operations, HMDs’ long-term impacts remain uncertain (Fang et al., 2019). Furthermore, the influence of language proficiency on HMD effectiveness needs exploration (Murauer et al, 2018). In order to fully comprehend the potential and limitations of HMDs, further research is necessary, targeting effective strategies for implementation and optimal AR user interface (UI) design. Objective: This study aims to evaluate an AR HMD system against traditional methods, focusing on its potential to aid non-native English-speaking warehouse workers and boost efficiency and accuracy in picking tasks. Our goal is to ascertain whether an AR aid system, utilizing universal and conformal design principles, can yield superior results in user performance, usability, and situational awareness compared to written instructions. Method: We identified the language-related challenges faced by foreign workers through interviews. Guided by these insights and Ganapathy’s mobile AR guidelines (Ganapathy et al., 2013), we designed an AR solution with universal symbols and intuitive interactions. The AR solution was prototyped using Microsoft HoloLens 2. To evaluate user experience with the proposed system, we conducted a within-subject experiment in a controlled laboratory environment, comparing this AR headset instruction with traditional written instructions. We employed the situation awareness rating technique (SART) questionnaire (Taylor et al., 2017) and system usability scale (SUS) questionnaire (Brooke et al., 1995), along with performance measures, to assess the effectiveness of the proposed system. Results: Our study with 17 participants indicated no significant difference in task completion time between traditional and AR headset instructions. However, AR significantly reduced package identification time ( M=6.89, SE=0.40 vs. M=10.15, SE=0.61). Moreover, people with the AR instructions had no errors while with the traditional written instructions had a total of 2 errors. The proposed AR aid also resulted in enhanced worker situation awareness by allowing them not to divide their attention across job instructions and the dynamic warehouse environment ( M=2.41, SE=0.24 vs. M=3.70, SE=0.35). The AR headset was perceived as easier to use ( M=4.35, SE=0.16 vs. M=3.35, SE=0.17) and better integrated various functions ( M=3.94, SE=0.16 vs. M=2.70, SE=0.19), despite some participants reported a need for technical assistance. Conclusion: The human-subject experiment demonstrates that the proposed AR aid system is effective in eliminating errors, improving ease of use, and enhancing situation awareness of foreign workers in warehouses. This study also underscores the importance of a user-centered approach in leveraging technology for users in diverse contexts. Application: Our proposal holds promising prospects beyond the scope of the study. Its potential extends to various safety- critical domains, including transportation, construction, and military operations, where operators’ awareness of the dynamic environment is crucial (Kim et al., 2020).

Augmented reality for industrial quality inspection: An experiment assessing task performance and human factors

Augmented reality (AR) technologies promise to increase the flexibility and productivity of the workforce by providing real-time information to workers right where it is needed. Following Cognitive Load Theory and Attention Theory, AR assistance can reduce the burden on workers’ mental capacity while performing a task, thereby improving task performance. The benefits of AR on industrial activities like assembly or maintenance have been investigated extensively, but the literature on AR-assisted quality control, specifically, quality inspection, is scarce. This stands in contrast to the importance of industrial quality inspection and highlights the need for research on the topic. In this work, we develop an AR-based system for head-mounted displays (HMDs) that visualizes product defects directly on physical products. We investigate its effect on task performance and human factors through an experiment. Participants performed multiple quality inspections using real manufacturing products and equipment with the help of an AR HMD, a screen, or no additional help. We find increased task performance using the AR HMD system. In comparison to screen-based assistance, a moderating effect of task difficulty was observed. Specifically, AR was more beneficial to task completion times given a difficult task. Furthermore, the AR HMD system reduced mental workload and received a positive rating regarding user experience. Our work contributes to the research in industrial AR usage through a novel AR HMD system for quality inspection and provides so far missing empirical and theoretically grounded evidence on its benefits with regard to task performance and human factors.

Exploring Gaze-assisted and Hand-based Region Selection in Augmented Reality

Region selection is a fundamental task in interactive systems. In 2D user interfaces, users typically use a rectangle selection tool to formulate a region using a mouse or touchpad. Region selection in 3D spaces, especially in Augmented Reality (AR) Head-Mounted Displays (HMDs) is different and challenging because users need to select an intended region via freehand mid-air gestures or eye-based actions that are touchless interactions. In this work, we aim to fill in the gap in the design of region selection techniques in AR HMDs. We first analyzed and discretized the interaction procedure of region selection and explored design possibilities for each step. We then developed four techniques for region selection in AR HMDs, which leveraged users' hand and gaze for unimodal or multimodal interaction. The techniques were evaluated via a user study with a controlled region selection task. The findings led to three design recommendations and two proof-of-concept application examples.

Usability Assessments for Augmented Reality Head-Mounted Displays in Open Surgery and Interventional Procedures: A Systematic Review

Augmented reality (AR) head-mounted displays (HMDs) are an increasingly popular technology. For surgical applications, the use of AR HMDs to display medical images or models may reduce invasiveness and improve task performance by enhancing understanding of the underlying anatomy. This technology may be particularly beneficial in open surgeries and interventional procedures for which the use of endoscopes, microscopes, or other visualization tools is insufficient or infeasible. While the capabilities of AR HMDs are promising, their usability for surgery is not well-defined. This review identifies current trends in the literature, including device types, surgical specialties, and reporting of user demographics, and provides a description of usability assessments of AR HMDs for open surgeries and interventional procedures. Assessments applied to other extended reality technologies are included to identify additional usability assessments for consideration when assessing AR HMDs. The PubMed, Web of Science, and EMBASE databases were searched through September 2022 for relevant articles that described user studies. User assessments most often addressed task performance. However, objective measurements of cognitive, visual, and physical loads, known to affect task performance and the occurrence of adverse events, were limited. There was also incomplete reporting of user demographics. This review reveals knowledge and methodology gaps for usability of AR HMDs and demonstrates the potential impact of future usability research.

Electroencephalogram-based mental workload prediction for using Augmented Reality head mounted display in construction assembly: A deep learning approach

Augmented Reality Head-Mounted Displays (AR HMD) are considered a promising technology for assisting on-site construction tasks such as assembly. However, concerns about creating information overload and becoming a distraction to workers outweigh the potential benefits to productivity and task performance. Previous research investigated workers' mental workloads as impacted by AR-based displays for assembly tasks. Within that context, this paper proposes a Long Short-Term Memory (LSTM) based approach to predict mental workload when using AR HMD for construction assembly, to forecast users' cognitive status under such complex working condition. Thirty participants were recruited to finish a wood frame assembly using an AR HMD projecting a 3D conformal model on the workspace. The proposed method provided a reliable prediction of real-time electroencephalogram (EEG) signal and mental workload. The outcomes validated the feasibility of LSTM model for EEG-based mental workload prediction and specifically provided a usable method for evaluating AR HMD use in construction tasks.