Reality Head Research Articles

Catch-Up Saccades in Vestibulo-Ocular Reflex Deficit: Contribution of Visual Information?

Catch-up saccades help to compensate for loss of gaze stabilization during rapid head rotation in case of vestibular deficit. While overt saccades observed after head rotation are obviously visually guided, some of these catch-up saccades occur with shorter latency while the head is still moving, anticipating the needed final eye position. These covert saccades seem to be generated based on the integration of multisensory inputs. Vision could be one of these inputs, but the known delay for triggering visually guided saccades questions this possibility. The main objective of this study is to evaluate the potential role of visual information for controlling (triggering and guiding) the first catch-up saccades in patients suffering from bilateral vestibulopathy. To investigate this, we used head impulse test in a virtual reality setting allowing to create different visuo-vestibular mismatch conditions. Twelve patients with bilateral vestibulopathy were recruited. We assessed in our patient group the validity of our virtual reality head impulse testing approach by comparing recorded eye and head movement to classical video head impulse test. Then, using the virtual reality system, we tested head impulse test under both normal and three visuo-vestibular mismatch conditions. In these mismatch conditions, the movement of the visual scene relative to the head movement was altered: decreased in amplitude by 50% (half), nullified (freeze), or inverted in direction (inverse). Recorded eye and head movements during these different conditions were then analyzed, more specifically the characteristics of the first catch-up saccade. Impaired vestibulo-ocular reflex required subjects to systematically perform catch-up saccades, which could be covert or overt. The latency of the first catch-up saccade increased along with the amount of visuo-vestibular mismatch between the four conditions (i.e., from normal to half to freeze to inverse) and, consequently, the mean percentage of covert saccades decreased with increasing visual feedback error. However, the freeze and inverse conditions allowed us to reveal the existence of many saccades performed in the wrong direction relative to visual feedback. These visually discordant saccades were present in over half of the trials, they were mainly covert and their percentage was inversely correlated with residual vestibulo-ocular reflex gain. Visual information significantly impacts catch-up saccade latency and the relative number of covert saccades during head impulse testing in vestibular deficit. However, in more than 50% of trials involving a visuo-vestibular mismatch, catch-up saccades remained directed in the compensatory direction relative to head movement, that is, they were visually discordant. Therefore, contrary to previously published proposals, visual information does not appear to be the primary component of the multisensory inputs required for the production of catch-up covert saccades in vestibular deficit. Finally, we discuss a new nomenclature of catch-up saccades in case of vestibular deficit introducing the terms of open and closed loop saccades.

Validation of an Augmented Reality Based Functional Method to Determine and Render the Hip Rotation Centre During Total Hip Arthroplasty.

We present a method to determine and visualise the functional centre of rotation (FCOR) of the hip during total hip arthroplasty using an augmented reality head mounted display (AR-HMD). We developed software allowing a HoloLens to provide inside-out infrared tracking of markers affixed to cadaver femurs and 3D printed acetabuli. Two observers rotated 20 cadaver femurs twice in a matching cup, producing 80 measurements. The FCOR of the acetabulum and femoral head was determined based on the point cloud generated from the displacement of the femoral trackers to the acetabular tracker. Compared to the ground truth, the FCOR resulted in an absolute error of 2.9±1.4mm for the acetabulum and 2.9±1.2mm for the femur, with 95th percentiles below 5.6 and 4.7mm. The proposed AR-HMD system offers an accurate and reproducible way to determine the femoral and acetabular FCOR in an experimental setting.

Eating Food Produced in Harmful Ways: Wrongful Complicity or Moral Tragedy?

Abstract Imagine you come across a leftover turkey sandwich that will expire soon, no one is around to eat it, and you cannot donate it to someone else before it goes bad. A new omnivore says that you should eat the sandwich. A common rebuttal to new omnivorism says that by eating the turkey sandwich you become complicit in industrialized animal agriculture: consuming the leftover sandwich amounts to a retroactive endorsement of the institution that produced it. This is the complicity objection to new omnivorism. In this paper, we argue that the complicity objection fails to ethically distinguish new omnivores and vegans. We suggest that we accept that there is no harmless-to-animals diet available, and so, rather than trying to find the impossible, we should confront this morally tragic reality head on. Doing so may support new omnivorism over veganism.

MIXED-REALITY GUIDANCE FOR PELVIC OSTEOTOMY AND ACETABULAR REALIGNMENT: PROOF OF CONCEPT

Mixed Reality has the potential to improve accuracy and reduce required dissection for the performance of peri-acetabular osteotomy. The current work assesses initial proof of concept of MR guidance for PAO.A PAO planning module, based on preoperative computed tomography (CT) imaging, allows for the planning of PAO cut planes and repositioning of the acetabular fragment. 3D files (holograms) of the cut planes and native and planned acetabulum positions are exported with the associated spatial information. The files are then displayed on mixed reality head mounted device (HoloLens2, Microsoft) following intraoperative registration using an FDA-cleared mixed reality application designed primary for hip arthroplasty (HipInsight). PAO was performed on both sides of a bone model (Pacific Research). The osteotomies and acetabular reposition were performed in accordance with the displayed holograms. Post-op CT imaging was performed for analysis. Cutting plane-accuracy was evaluated using a best-fit plane and 2D angles (°) between the planned and achieved supra (SA)- and retroacetabular (RA) osteotomy and retroacetabular and ischial osteotomies (IO) were measured.To evaluate the accuracy of acetabular reorientation, we digitized the acetabular rim and calculated the acetabular opening plane. Absolute errors of planned and achieved operative inclination and anteversion (°) of the acetabular fragment, as well as 3D lateral-center-edge (LCE) angles were calculated.The mean absolute difference between the planned and performed osteotomy angles was 3 ± 3°.The mean absolute error between planned and achieved operative anteversion and inclination was 1 ± 0° and 0 ± 0° respectively. Mean absolute error between planned and achieved 3D LCE angle was 0.5 ± 0.7°.Mixed-reality guidance for the performance of pelvic osteotomies and acetabular fragment reorientation was feasible and highly accurate. This solution may improve the current standard of care by enabling reliable and precise reproduction of the desired acetabular realignment.

Mixed reality head mounted displays for enhanced indoor point cloud segmentation with virtual seeds

Mixed Reality (MR) Head Mounted Displays (HMDs) offer a hitherto underutilized set of advantages compared to conventional 3D scanners. These benefits, inherent to MR-HMDs albeit not originally intended for such applications, encompass the freedom of hand movement, hand tracking capabilities, and real-time mesh visualization. This study leverages these attributes to enhance indoor scanning process. The primary innovation lies in the conceptualization of manual-positioned MR virtual seeds for the purpose of indoor point cloud segmentation via a region-growing approach. The proposed methodology is effectively implemented using the HoloLens 2 platform. An application is designed to enable the remote placement of virtual tags based on the user's visual focus on the MR-HMD display. This non-intrusive interface is further enriched with expedited tag saving and deletion functionalities, as well as augmented tag visualization through overlaying them on real-world objects. To assess the practicality of the proposed method, a comprehensive real-world case study spanning an area of 330 s2 is conducted. Remarkably, the survey demonstrates remarkable efficiency, with 20 virtual tags swiftly deployed, each requiring a mere 2 s for precise positioning. Subsequently, these virtual tags are employed as seeds in a region-growing algorithm for point cloud segmentation. The accuracy of virtual tag positioning is found to be exceptional, with an average error of 2.4 ± 1.8 cm. Importantly, the user experience is significantly enhanced, leading to improved seed positioning and, consequently, more accurate final segmentation results.

A Pipeline for Characterising Virtual Reality Head Mounted Displays

A Pipeline for Characterising Virtual Reality Head Mounted Displays

P‐49: Angular Dependent Point Spread Function Analysis for Virtual Reality Head Mounted Displays

We demonstrate a point spread function method for measuring the angular dependence of the spatial resolution of a virtual reality head mounted display. The angular dependence of spatial resolution of the imaging detector was characterized to remove the detector's contribution at each angular location measured on the VR HMD. This method allows measuring the twodimensional spatial resolution and modulation transfer function.

Overground walking while using a virtual reality head mounted display increases variability in trunk kinematics and reduces dynamic balance in young adults

AbstractThis study analyzed the effects of walking freely in virtual reality (VR) compared to walking in the real-world on dynamic balance and postural control. For this purpose, nine male and twelve female healthy participants underwent standard 3D gait analysis while walking randomly in a real laboratory and in a room-scale overground VR environment resembling the real laboratory. The VR was delivered to participants by a head-mounted-display which was operated wirelessly and calibrated to the real-world. Dynamic balance and postural control were assessed with (1) the margin of stability (MOS) in the anteroposterior (AP-MOS) and mediolateral (ML-MOS) directions at initial-contact, (2) the relationship between the mediolateral center of mass (COM) position and acceleration at mid-stance with subsequent step width, (3) and trunk kinematics during the entire gait cycle. We observed increased mediolateral (ML) trunk linear velocity variability, an increased coupling of the COM position and acceleration with subsequent step width, and a decrease in AP-MOS while walking in VR but no change in ML-MOS when walking in VR. Our findings suggest that walking in VR may result in a less reliable optical flow, indicated by increased mediolateral trunk kinematic variability, which seems to be compensated by the participants by slightly reweighing sensorimotor input and thereby consciously tightening the coupling between the COM and foot placement to avoid a loss of balance. Our results are particularly valuable for future developers who want to use VR to support gait analysis and rehabilitation.

Augmented Reality Order Picking Aid for Foreign Workers in Warehouses

Background: In warehouse logistics, picking work comprises roughly 60% of the costs, emphasizing the need for efficiency (Matsumoto et al., 2019). Traditionally, workers utilize written instructions, which can be challenging for those less language proficient. The integration of augmented reality (AR) head- mounted displays (HMDs) may enhance accuracy and efficiency (Matsumoto et al., 2019). However, AR HMDs may come with some drawbacks—after extended use, some users report discomfort and decreased performance (Vidovič & Gajšek, 2020). Despite promising improvements in warehouse operations, HMDs’ long-term impacts remain uncertain (Fang et al., 2019). Furthermore, the influence of language proficiency on HMD effectiveness needs exploration (Murauer et al, 2018). In order to fully comprehend the potential and limitations of HMDs, further research is necessary, targeting effective strategies for implementation and optimal AR user interface (UI) design. Objective: This study aims to evaluate an AR HMD system against traditional methods, focusing on its potential to aid non-native English-speaking warehouse workers and boost efficiency and accuracy in picking tasks. Our goal is to ascertain whether an AR aid system, utilizing universal and conformal design principles, can yield superior results in user performance, usability, and situational awareness compared to written instructions. Method: We identified the language-related challenges faced by foreign workers through interviews. Guided by these insights and Ganapathy’s mobile AR guidelines (Ganapathy et al., 2013), we designed an AR solution with universal symbols and intuitive interactions. The AR solution was prototyped using Microsoft HoloLens 2. To evaluate user experience with the proposed system, we conducted a within-subject experiment in a controlled laboratory environment, comparing this AR headset instruction with traditional written instructions. We employed the situation awareness rating technique (SART) questionnaire (Taylor et al., 2017) and system usability scale (SUS) questionnaire (Brooke et al., 1995), along with performance measures, to assess the effectiveness of the proposed system. Results: Our study with 17 participants indicated no significant difference in task completion time between traditional and AR headset instructions. However, AR significantly reduced package identification time ( M=6.89, SE=0.40 vs. M=10.15, SE=0.61). Moreover, people with the AR instructions had no errors while with the traditional written instructions had a total of 2 errors. The proposed AR aid also resulted in enhanced worker situation awareness by allowing them not to divide their attention across job instructions and the dynamic warehouse environment ( M=2.41, SE=0.24 vs. M=3.70, SE=0.35). The AR headset was perceived as easier to use ( M=4.35, SE=0.16 vs. M=3.35, SE=0.17) and better integrated various functions ( M=3.94, SE=0.16 vs. M=2.70, SE=0.19), despite some participants reported a need for technical assistance. Conclusion: The human-subject experiment demonstrates that the proposed AR aid system is effective in eliminating errors, improving ease of use, and enhancing situation awareness of foreign workers in warehouses. This study also underscores the importance of a user-centered approach in leveraging technology for users in diverse contexts. Application: Our proposal holds promising prospects beyond the scope of the study. Its potential extends to various safety- critical domains, including transportation, construction, and military operations, where operators’ awareness of the dynamic environment is crucial (Kim et al., 2020).

Hand interaction designs in mixed and augmented reality head mounted display: a scoping review and classification

Mixed reality has made its first step towards democratization in 2017 with the launch of a first generation of commercial devices. As a new medium, one of the challenges is to develop interactions using its endowed spatial awareness and body tracking. More specifically, at the crossroad between artificial intelligence and human-computer interaction, the goal is to go beyond the Window, Icon, Menu, Pointer (WIMP) paradigm humans are mainly using on desktop computer. Hand interactions either as a standalone modality or as a component of a multimodal modality are one of the most popular and supported techniques across mixed reality prototypes and commercial devices. In this context, this paper presents scoping literature review of hand interactions in mixed reality. The goal of this review is to identify the recent findings on hand interactions about their design and the place of artificial intelligence in their development and behavior. This review resulted in the highlight of the main interaction techniques and their technical requirements between 2017 and 2022 as well as the design of the Metaphor-behavior taxonomy to classify those interactions.

Electroencephalogram-based mental workload prediction for using Augmented Reality head mounted display in construction assembly: A deep learning approach

Augmented Reality Head-Mounted Displays (AR HMD) are considered a promising technology for assisting on-site construction tasks such as assembly. However, concerns about creating information overload and becoming a distraction to workers outweigh the potential benefits to productivity and task performance. Previous research investigated workers' mental workloads as impacted by AR-based displays for assembly tasks. Within that context, this paper proposes a Long Short-Term Memory (LSTM) based approach to predict mental workload when using AR HMD for construction assembly, to forecast users' cognitive status under such complex working condition. Thirty participants were recruited to finish a wood frame assembly using an AR HMD projecting a 3D conformal model on the workspace. The proposed method provided a reliable prediction of real-time electroencephalogram (EEG) signal and mental workload. The outcomes validated the feasibility of LSTM model for EEG-based mental workload prediction and specifically provided a usable method for evaluating AR HMD use in construction tasks.

Echo-Endoscopy Combined with Virtual Reality: A Whole Perspective of Laparoscopic Common Bile Duct Exploration in Children

Introduction: Endoscopic procedures are performed more frequently in children due to technological advances that can be safely performed in an adequate setting with a support of a multidisciplinary team. Pediatric indications for ERCP (endoscopic retrograde cholangiopancreatography) and EUS (endoscopic ultrasound) occur mainly due to congenital malformations. In a pediatric case series, we report the application of EUS combined with duodenoscopy, eventually associated with ERCP and minimally invasive surgery, highlighting the importance of defining a tailored dedicated management pathway for each patient. Patients and methods: A series of 12 patients, managed at our Center in the last three years, were evaluated, and their management was discussed. Results: EUS was performed in eight patients and permitted the differential diagnosis of duplication cysts and the visualization of the biliary tree and pancreatic anatomy. ERCP was attempted in five patients: in one case, it permitted the preservation of pancreatic tissue, postponing surgery and in three patients, it was technically unfeasible. MIS (minimally invasive surgery) was performed in seven patients, two with laparoscopic common bile duct exploration (LCBDE). Precise anatomical definition and the possibility of surgical simulation and team sharing were evaluated under VR HMD (Virtual Reality Head Mounted Display) in four cases. Conclusions: Exploration of the common bile duct in children differs from that of the adult population and combines echo-endoscopy and ERCP. The integrated use of minimally invasive surgery in the pediatric area is necessary for the whole management perspective in complex malformations and small patients. The introduction in the clinical practice of a preoperative study with Virtual Reality allows a better survey of the malformation and a tailored treatment.

The pupil near response is short lasting and intact in virtual reality head mounted displays.

The pupil of the eye constricts when moving focus from an object further away to an object closer by. This is called the pupil near response, which typically occurs together with accommodation and vergence responses. When immersed in virtual reality mediated through a head-mounted display, this triad is disrupted by the vergence-accommodation conflict. However, it is not yet clear if the disruption also affects the pupil near response. Two experiments were performed to assess this. The first experiment had participants follow a target that first appeared at a far position and then moved to either a near position (far-to-near; FN) or to another far position (far-to-far; FF). The second experiment had participants follow a target that jumped between five positions, which was repeated at several distances. Experiment 1 showed a greater pupil constriction amplitude for FN trials, compared to FF trials, suggesting that the pupil near response is intact in head-mounted display mediated virtual reality. Experiment 2 did not find that average pupil dilation differed when fixating targets at different distances, suggesting that the pupil near response is transient and does not result in sustained pupil size changes.

Consideration of blue light hazard for virtual reality head mounted displays

Virtual reality (VR) introduces new approaches to creating virtual spaces for empirical studies and design processes. At the same time, VR platforms for research and education purposes in the field of daylight, such as one being built at TU Berlin, have to consider technical limitations. Besides the imperfections caused by simulation tools, VR head mounted displays (VR HMDs) bring additional shortcomings, one of the most obvious being the low maximum luminance compared to daylight. However, considering a typical viewing distance within VR HMDs is about 15 mm, and a usage duration can last up to 8 hours or more, concerns have arisen about the safety of displays with high luminance levels, particularly as a prototype claiming a maximum luminance of up to 6 000 cd/m2 has been introduced in the last years. Therefore, this paper compares the blue light hazard (BLH)-weighted radiances of VR HMDs to given exposure limits provided by the ICNIRP guidelines on limits of exposure to incoherent visible and infrared radiation and by the IEC 62471 (photobiological safety of lamps and lamp systems). The paper concludes with a statement on the BLH safety of VR HMDs.

Augmented Reality Head Mounted Display based ultrasound navigation for spine surgery.

Augmented Reality Head Mounted Display based ultrasound navigation for spine surgery.

HoloMentor

The introduction of Augmented Reality (AR) Head Mounted Displays (HMDs) in collaboration between remote and local workers, introduces new challenges given that camera views are now mobile. We introduce HoloMentor, an AR HMD-based collaborative system designed for remote instruction over live mobile views during physical tasks. Through Actionport, we provide a world-stabilized area where remote helpers can dynamically place a pointer and annotations on the physical environment. Through Actionpad, we provide an indirect input mechanism with an absolute position to the Actionport. We show how these innovations worked for participants engaged in a remote instructional task and how they supported effective and efficient communication. Finally, we provide the next steps for addressing AR on mobile views for remote instruction.

124. A systematic review of techniques used to validate the registration of AR images using a head-mounted device to navigate surgery

Augmented Reality (AR) Head Mounted Devices (HMD) allow the wearer to have digital images superposed onto their field of vision. It is being used to superpose annotations onto the surgical field akin to a navigation system. This review examines the published validation studies on HMD-AR systems, their reported protocols and outcomes. The aim is to establish commonalities and an acceptable registration outcome.

Spatiotemporal image quality of virtual reality head mounted displays

Virtual reality (VR) head mounted displays (HMDs) require both high spatial resolution and fast temporal response. However, methods to quantify the VR image quality in the spatiotemporal domain when motion exists are not yet standardized. In this study, we characterize the spatiotemporal capabilities of three VR devices: the HTC VIVE, VIVE Pro, and VIVE Pro 2 during smooth pursuit. A spatiotemporal model for VR HMDs is presented using measured spatial and temporal characteristics. Among the three evaluated headsets, the VIVE Pro 2 improves the display temporal performance using a fast 120 Hz refresh rate and pulsed emission with a small duty cycle of 5%. In combination with a high pixel resolution beyond 2 k times 2 k per eye, the VIVE Pro 2 achieves an improved spatiotemporal performance compared to the VIVE and VIVE Pro in the high spatial frequency range above 8 cycles per degree during smooth pursuit. The result demonstrates that reducing the display emission duty cycle to less than 20% is beneficial to mitigate motion blur in VR HMDs. Frame rate reduction (e.g., to below 60 Hz) of the input signal compared to the display refresh rate of 120 Hz yields replicated shadow images that can affect the image quality under motion. This work supports the regulatory science research efforts in development of testing methods to characterize the spatiotemporal performance of VR devices for medical use.

Extended reality for mental health: Current trends and future challenges

Virtual and augmented reality have been used to diagnose and treat several mental health disorders for decades. Technological advances in these fields have facilitated the availability of commercial solutions for end customers and practitioners. However, there are still some barriers and limitations that prevent these technologies from being widely used by professionals on a daily basis. In addition, the COVID-19 pandemic has exposed a variety of new scenarios in which these technologies could play an essential role, like providing remote treatment. Disorders that traditionally had received less attention are also getting in the spotlight, such as depression or obsessive-compulsive disorder. Improvements in equipment and hardware, like Mixed Reality Head Mounted Displays, could help open new opportunities in the mental health field. Extended reality (XR) is an umbrella term meant to comprise Virtual reality (VR), mixed reality (MR), and augmented reality (AR). While XR applications are eminently visual, other senses are being explored in literature around multisensory interactions, such as auditory, olfactory, or haptic feedback. Applying such stimuli within XR experiences around mental disorders is still under-explored and could greatly enrich the therapeutic experience. This manuscript reviews recent research regarding the use of XR for mental health scenarios, highlighting trends, and potential applications as well as areas for improvement. It also discusses future challenges and research areas in upcoming topics such as the use of wearables, multisensory, and multimodal interaction. The main goal of this paper is to unpack how these technologies could be applied to XR scenarios for mental health to exploit their full potential and follow the path of other health technologies by promoting personalized medicine.

The effect of presentation position and multichannel approach on learning performance in the use of an augmented reality–head worn display interface for train-driving training

ABSTRACT The objective of this study was to determine the differential effects of the presentation position of the augmented reality–head worn display (AR-HWD) interface and the audiovisual-dominant multimodal learning material on learning performance and cognitive load across different learning tasks in training for high-speed train driving. We selected 48 participants to conduct an AR driving training experiment for high-speed trains. The results indicated that whether or not the AR interface was presented next to the learning target did not significantly affect learning performance and cognitive load. In complex learning tasks, such as learning to operate the controller of a high-speed train and to remember the steps of swapping the front and tail of the train, the auditory-led AR learning materials were effective in improving learning performance and reducing cognitive load compared with the visual-led AR learning materials. In the simple task of icon recognition, the visual-led AR learning materials performed better in terms of learning performance and cognitive load. The results of this study can guide the design of AR-HWD interfaces for different learning tasks in training for high-speed train driving, which can improve learning performance and reduce cognitive load during the learning process. HIGHLIGHTS We studied the position of AR interface presentation and AR multichannel learning for high-speed train-driving training. Whether or not the AR interface was presented next to the target did not have a significant effect on learning performance for that target. Auditory-dominant interfaces outperformed visual-dominant interfaces in complex learning tasks. Visual-dominant interfaces outperformed auditory-dominant interfaces in simple recognition tasks.