Mixed Reality Applications Research Articles

Mixed Reality for Cranial Neurosurgical Planning: A Single-Center Applicability Study With the First 107 Subsequent Holograms.

Mixed reality (MxR) benefits neurosurgery by improving anatomic visualization, surgical planning and training. We aim to validate the usability of a dedicated certified system for this purpose. All cases prepared with MxR in our center in 2022 were prospectively collected. Holographic rendering was achieved using an incorporated fully automatic algorithm in the MxR application, combined with contrast-based semiautomatic rendering and/or manual segmentation where necessary. Hologram segmentation times were documented. Visualization during surgical preparation (defined as the interval between finalized anesthesiological induction and sterile draping) was performed using MxR glasses and direct streaming to a side screen. Surgical preparation times were compared with a matched historical cohort of 2021. Modifications of the surgical approach after 3-dimensional (3D) visualization were noted. Usability was assessed by evaluating 7 neurosurgeons with more than 3 months of experience with the system using a Usefulness, Satisfaction and Ease of use (USE) questionnaire. One hundred-seven neurosurgical cases prepared with a 3D hologram were collected. Surgical indications were oncologic (63/107, 59%), cerebrovascular (27/107, 25%), and carotid endarterectomy (17/107, 16%). Mean hologram segmentation time was 39.4 ± 20.4 minutes. Average surgical preparation time was 48.0 ± 17.3 minutes for MxR cases vs 52 ± 17 minutes in the matched 2021 cohort without MxR (mean difference 4, 95% CI 1.7527-9.7527). Based on the 3D hologram, the surgical approach was modified in 3 cases. Good usability was found by 57% of the users. The perioperative use of 3D holograms improved direct anatomic visualization while not significantly increasing intraoperative surgical preparation time. Usability of the system was adequate. Further technological development is necessary to improve the automatic algorithms and reduce the preparation time by circumventing manual and semiautomatic segmentation. Future studies should focus on quantifying the potential benefits in teaching, training, and the impact on surgical and functional outcomes.

INNOVATION IN AGING & ACQUIRED BRAIN INJURY REHABILITATION: THE POTENTIAL OF MIXED REALITY TECHNOLOGIES

Abstract Background Every 21 seconds an American experiences a Traumatic Brain Injury (TBI), and every 40 seconds, another endures a stroke. These events fall under Acquired Brain Injuries (ABI) with prevalence rates impacted by an aging population. ABI can lead to various physical, cognitive, and mental impairments, emphasizing the need for innovative rehabilitation strategies, including those specific to older adults. Mixed Reality (MR) technologies offer potential in enhancing ABI rehabilitation, yet face challenges such as methodological inconsistencies, differing clinical populations, and exaggerated efficacy claims. Objectives 1) Review MR’s role in ABI rehabilitation among older adults, assessing the implications of aging, clinical, and technological considerations. 2) Present ongoing MR research at the Glenrose Rehabilitation Hospital (GRH, Edmonton, Canada). Methods 1) A systematic review following PRISMA guidelines was performed across seven databases, with two independent reviewers analyzing the data. The analysis emphasized clinical objectives, MR systems, levels of evidence, and technology readiness levels. 2) QR codes that link to videos will highlight the ongoing research and development of MR-delivered ABI rehabilitation at the GRH. Results Twenty-six studies met the inclusion criteria, totalling 453 subjects with ABI (mean age: 60 ± 5.34 years). MR applications mainly targeted upper limb motor rehabilitation, revealing an overall low level of evidence and a median technology readiness level of 6 (prototypes tested in relevant environments). Conclusion Despite existing variability and technological challenges, the promising results stress the importance of ongoing research and innovation in MR rehabilitation. The GRH stands as a key research hub, actively advancing this field.

Fast scaled cylindrical holography based on scaled convolution

Recently, the 360°display of cylindrical holography has garnered significant attention. However, existing studies have been limited to concentric cylindrical surfaces with equal heights, which restricts the height of the objects. In this paper, a fast scaled cylindrical holography is proposed based on scaled convolution to break this constraint. Firstly, the scale cylindrical diffraction is derived using the Rayleigh–Sommerfeld diffraction formula for scaled cylindrical holography, in which the object’s height can be adjusted. Then, a method of fast scaled cylindrical diffraction is presented by scaled convolution based on scaled Fourier Transform, and it significantly reduces computational time, enhancing its practical feasibility. Furthermore, the proposed method is extended to 3D objects using a layer-oriented approach. Moreover, the effectiveness and feasibility of the proposed method are verified through numerical simulations and optical experiments. Therefore, the issue of the constraint on the object’s height in cylindrical holography is thoroughly discussed and effectively addressed for the first time. This breakthrough paves the way for the development of a zoom-able cylindrical holographic display with immense potential for applications in mixed reality and beyond.

CONTINUOUS BIM ALIGNMENT FOR MIXED REALITY VISUALISATION

Abstract. Several methods exist that can be used to perform initial alignment of Building information models (BIMs) to the real building for Mixed Reality (MR) applications, such as marker-based or markerless visual methods, but this alignment is susceptible to drift over time. The existing model-based methods that can be used to maintain this alignment have multiple limitations, such as the use of iterative processes and poor performance in environments with either too many or not enough lines. To address these issues, we propose an end-to-end trainable Convolutional Neural Network (CNN) that takes a real and synthetic BIM image pair as input to regress the 6 DoF relative camera pose difference between them directly. By correcting the relative pose error we are able to considerably improve the alignment of the BIM to the real building. Furthermore, the results of our experiments demonstrate good performance in a challenging environment and high resilience to domain shift between synthetic and real images. A high localisation accuracy of approximately 7.0 cm and 0.9° is achieved which indicates the method can be used to reduce the camera tracking drift for MR applications.

Application of mixed reality and artificial intelligence to assist medical students in learning injection technique

Introduction: The usage of immersive technology has advanced in a number of areas of life because of the development of technology that keeps pace with the times. Another immersive technology that combines VR and AR is mixed reality (MR), which enables us to interact with 3-dimensional objects in the real world. Since MR technology gives a more nuanced experience, the market is highly promising. This study aims to evaluate the application of mixed reality and artificial intelligence to assist medical students in learning injection technique. Methods: This type of research is analytic with a quantitative and qualitative approach to prove the purpose of the research. This research involved 40 students. Due to the creative nature of immersive technology, it must be combined with other technologies to produce an even more complicated and engaging experience. In order to enhance the quality of the user experience, we will merge MR immersive technology with AI in this research for medical educational field. The integration of these two technologies via an application that can be launched on a Hololens 2 and Magic Leap 1 device and can identify person in a laboratory to support in student learning. Results: For instance, students can utilize artificial intelligence (AI) to learn the names of objects in the lab and do simulation about injection technique. The study 's outcomes are presented in software testing (FPS, CPU, GPU, and load scene) an in the form of user testing utilizing the PIECES Framework (Performance, Information and Data, Economy, Control and Security, Efficiency, and Service), which evaluates the application's utility or significance as well as the satisfaction of its users. Conclusion: The system was able to develop a combining application of artificial intelligence and mixed reality for detecting objects in laboratories to assist learning students, according to the study's conclusions.

Enhanced Precision in Genioplasty: A Novel Intraoperative Spatial Repositioning Using Computer-Aided Design and Manufacturing Technology and a Holographic Mixed Reality Application

Genioplasty is performed for the orthognathic surgical correction of dentofacial deformities. This article reports a safe and accurate method for genioplasty combining a novel three-dimensional (3D) device with mixed reality (MR)-assisted surgery using a registration marker and a head-mounted display. Four types of devices were designed based on the virtual operation: a surgical splint with a connector; an osteotomy device; a repositioning device; and a registration marker. Microsoft HoloLens 2 and Holoeyes MD were used to project holograms created using computed tomography (CT) data onto the surgical field to improve the accuracy of the computer-aided designed and manufactured (CAD/CAM) surgical guides. After making an incision on the oral vestibule, the splint was fitted on the teeth and the osteotomy device was mounted at the junction site, placed directly on the exposed mandible bone surface. Temporary screws were fixed into the screw hole. An ultrasonic cutting instrument was used for the osteotomy. After separating the bone, a repositioning device was connected to the splint junction and bone segment, and repositioning was performed. At the time of repositioning, the registration marker was connected to the splint junction, and mandible repositioning was confirmed three-dimensionally through HoloLens 2 into the position specified in the virtual surgery. The rate of overlay error between the preoperative virtual operation and one-month postoperative CT data within 2 mm was 100%. CAD/CAM combined with MR enabled accurate genioplasty.

Light-Occlusion Text Entry in Mixed Reality

Text entry is a recurring task in mixed reality (MR) applications, and the ability of eyes-free text entry methods to allow users to enter text without focusing on the input device is ideal and compelling. However, existing eyes-free text entry methods leave much to be desired regarding efficiency and accuracy. In this paper, we propose a new light-occlusion text entry method in MR environment that uses dual thumb typing on a touchscreen. We design a partially visible keyboard as visual feedback to improve user performance. In addition, we optimize the underlying keyboard by collecting eyes-free typing data through a user study. The results show that our method has high typing speed, low error rate, and is very novice-friendly. After a short training period, the average typing speed of the novice group can reach 26.23 WPM (words per minute), while the average typing speed of the potential expert group can reach 30.62 WPM.

Creating Location-Based Augmented Reality Games and Immersive Experiences for Touristic Destination Marketing and Education

The aim of this paper is to present an approach that utilizes several mixed reality technologies for touristic promotion and education. More specifically, mixed reality applications and games were created to promote the mountainous areas of Western Macedonia, Greece, and to educate visitors on various aspects of these destinations, such as their history and cultural heritage. Location-based augmented reality (AR) games were designed to guide the users to visit and explore the destinations, get informed, gather points and prizes by accomplishing specific tasks, and meet virtual characters that tell stories. Furthermore, an immersive lab was established to inform visitors about the region of interest through mixed reality content designed for entertainment and education. The lab visitors can experience content and games through virtual reality (VR) and augmented reality (AR) wearable devices. Likewise, 3D content can be viewed through special stereoscopic monitors. An evaluation of the lab experience was performed with a sample of 82 visitors who positively evaluated features of the immersive experience such as the level of satisfaction, immersion, educational usefulness, the intention to visit the mountainous destinations of Western Macedonia, intention to revisit the lab, and intention to recommend the experience to others.

Collaborative diagnosis in mixed-reality using deep-learning networks and RE-WAPICP algorithm

This investigation explores the use of mixed-reality in collaborative diagnosis by sharing medical data in real-time between multiple physicians using Head-Mounted Display (HMD) devices. Object detection and alignment of the digitized data with the object are the backbone in any mixed-reality application. In this paper, deep-learning networks are used in detecting the patient’s face in the physical world and the medical data is aligned to the patient via the Region-Enhanced-Weight-and-Perturb Iterative-Closest-Point (RE-WAPICP) algorithm. Experiments were performed by sharing a 3D digital model of intracerebral vascular with multi-viewers in a mix-reality environment and the results show that this approach is feasible.

Virtual, augmented, and mixed reality as a versatile tool in food consumer behavior evaluation: Recent advances in aroma, taste, and texture incorporation.

The perceptual behavior of consumers on a product displayed in the market has a vital role in analyzing the importance given to that product. Therefore, various strategies have been developed to understand this consumer behavior in the selection of products. Immersive technologies like virtual, augmented, and mixed reality are among them. With the foremost feature of immersion in the virtual world and interaction of users with virtual objects, virtual reality, and augmented reality have unlocked their potential in research and a user-friendly tool for analyzing consumer behavior. In addition to these technologies, mixed reality also has a significant role in investigating consumer behavior. Studies on immersive technologies in food applications are vast, hence this review focuses on the applications of virtual, augmented, and mixed reality in the food selection behavior of consumers. The behavioral studies are elicited to develop new products based on consumer needs, to understand the shopping behavior in supermarkets for real-time usage, and to know the influence of emotions in a selection of products. The findings suggest that virtual, augmented, and mixed reality induce immersion of the users in food selection behavioral studies. Information on the technological advancements in the tools used for bringing immersion and interaction are discussed for its futuristic applications in food. Though immersive technology gives users a realistic virtual environment experience, its application in food systems is in the budding stage. More research on human response studies would contribute to its innovative and inevitable application in the future.

Application of virtual and mixed reality for 3D visualization in intracranial aneurysm surgery planning: a systematic review.

Precise preoperative anatomical visualization and understanding of an intracranial aneurysm (IA) are fundamental for surgical planning and increased intraoperative confidence. Application of virtual reality (VR) and mixed reality (MR), thus three-dimensional (3D) visualization of IAs could be significant in surgical planning. Authors provide an up-to-date overview of VR and MR applied to IA surgery, with specific focus on tailoring of the surgical treatment. A systematic analysis of the literature was performed in accordance with the PRISMA guidelines. Pubmed, and Embase were searched to identify studies reporting use of MR and VR 3D visualization in IA surgery during the last 25 years. Type and number of IAs, category of input scan, visualization techniques (screen, glasses or head set), inclusion of haptic feedback, tested population (residents, fellows, attending neurosurgeons), and aim of the study (surgical planning/rehearsal, neurosurgical training, methodological validation) were noted. Twenty-eight studies were included. Eighteen studies (64.3%) applied VR, and 10 (35.7%) used MR. A positive impact on surgical planning was documented by 19 studies (67.9%): 17 studies (60.7%) chose the tailoring of the surgical approach as primary outcome of the analysis. A more precise anatomical visualization and understanding with VR and MR was endorsed by all included studies (100%). Application of VR and MR to perioperative 3D visualization of IAs allowed an improved understanding of the patient-specific anatomy and surgical preparation. This review describes a tendency to utilize mostly VR-platforms, with the primary goals of a more accurate anatomical understanding, surgical planning and rehearsal.

User experience of mixed reality applications for healthy ageing: A systematic review

Mixed reality (MR) technologies are being used increasingly to support healthy ageing, but past reviews have concentrated on the efficacy of the technology. This systematic review provides a synthesis of recent experimental studies on the instrumental, emotional and non-instrumental aspects of user experience of healthy older adults in relation to MR-related applications. The review was listed on PROSPERO, utilised a modified PICOS framework, and canvassed all published work between January 2010 to July 2021 that appeared in major databases (Scopus, PubMed, CINAHL, Web of Science, and the Cochrane Library). The literature search revealed 15 eligible studies. Results indicated that all included studies measured the instrumental quality of their applications, all but two studies measured the emotional reactions triggered by gameplay, and only six studies examined participants’ perception of non-instrumental quality of the applications. All included studies focused on improving a health domain such as cognitive or physical training. This suggests that the instrumental quality of the MR applications remains the focus of user experience studies, with far fewer studies examining the non-instrumental quality of the applications. Implications for game design and future research are discussed.

Enhancing Campus Familiarity: A Comparative Study of Augmented and Mixed Reality Applications for UMSIDA Campus 2


 
 
 
 
 
 
 This study addresses the need for improved orientation among Muhammadiyah University of Sidoarjo (UMSIDA) students, particularly newcomers unfamiliar with the layout of Campus 2. Leveraging the Multimedia Development Life Cycle (MDLC) Method, researchers developed and compared Augmented Reality (AR) and Mixed Reality (MR) applications, utilizing Faculty of Science and Technology brochures to render 3D representations of UMSIDA Campus 2 buildings. An online questionnaire garnered feedback from 25 UMSIDA students, with 88% expressing approval for the AR-based introduction application. To better assess its impact on prospective student enrollment, PMB Team members were consulted, resulting in 66.7% favoring the MR application for its enhanced engagement. Notably, both AR and MR applications were deemed equally useful by 66.7% of respondents. Consequently, these applications will be integrated into a comprehensive solution for further evaluation and optimization. This research underscores the potential of immersive technologies in enhancing campus navigation and highlights the significance of user preference in educational applications of augmented and mixed reality.
 Highlight:
 
 Utilizing MDLC Method: The study leverages the Multimedia Development Life Cycle (MDLC) Method for the development and comparison of Augmented Reality (AR) and Mixed Reality (MR) applications, targeting improved campus orientation for UMSIDA students.
 User Preference in Immersive Technologies: Feedback from UMSIDA students and PMB Team members revealed a preference for the Mixed Reality (MR) application, emphasizing the significance of user preference in educational applications of augmented and mixed reality.
 Integration for Enhanced Effectiveness: The AR and MR applications will be integrated into a comprehensive solution, highlighting the commitment to ongoing evaluation and optimization to further enhance campus navigation for UMSIDA students.
 
 Keyword: Orientation, Augmented Reality, Mixed Reality, Campus Navigation, Immersive Technologies
 
 
 
 
 
 

Enhanced Wearable Force-Feedback Mechanism for Free-Range Haptic Experience Extended by Pass-Through Mixed Reality

We present an extended prototype of a wearable force-feedback mechanism coupled with a Meta Quest 2 head-mounted display to enhance immersion in virtual environments. Our study focuses on the development of devices and virtual experiences that place significant emphasis on personal sensing capabilities, such as precise inside-out optical hand, head, and controller tracking, as well as lifelike haptic feedback utilizing servos and vibration rumble motors, among others. The new prototype addresses various limitations and deficiencies identified in previous stages of development, resulting in significant user performance improvements. Key enhancements include weight reduction, wireless connectivity, optimized power delivery, refined haptic feedback intensity, improved stylus alignment, and smooth transitions between stylus use and hand-tracking. Furthermore, the integration of a mixed reality pass-through feature enables users to experience a comprehensive and immersive environment that blends physical and virtual worlds. These advancements pave the way for future exploration of mixed reality applications, opening up new possibilities for immersive and interactive experiences that combine useful aspects of real and virtual environments.

Mixed reality training application to perform obstetric pulsed-wave Doppler ultrasound

Pulsed-wave Doppler ultrasound is a widely used technique for monitoring pregnancies. As ultrasound equipment becomes more advanced, it becomes harder to train practitioners to be proficient in the procedure as it requires the presence of an expert, access to high-tech equipment as well as several volunteering patients. Immersive environments such as mixed reality can help trainees in this regard due to their capabilities to simulate real environments and objects. In this article, we propose a mixed reality application to facilitate training in performing pulsed-wave Doppler ultrasound when acquiring a spectrogram to measure blood velocity in the umbilical cord. The application simulates Doppler spectrograms while the trainee has the possibility of adjusting parameters such as pulse repetition frequency, sampling depth, and beam-to-flow angle. This is done using a combination of an optimized user interface, 3D-printed objects tracked using image recognition and data acquisition from a gyroscope. The application was developed for Microsoft HoloLens as the archetype of mixed reality, while a 3D-printed abdomen was used to simulate a patient. The application aims to aid in both simulated and real-life ultrasound procedures. Expert feedback and user-testing results were collected to validate the purpose and use of the designed application. Design science research was followed to propose the intended application while contributing to the literature on leveraging immersive environments for medical training and practice. Based on the results of the study, it was concluded that mixed reality can be efficiently used in ultrasound training.

Mixed reality and the Internet of Things: Bridging the virtual with the real

As Mixed Reality applications penetrate people's daily activities, strong synergies with Geospatial technologies and the Internet of Things are revealed. The first is due to the significance of the spatial reference of all involved actors of a Mixed Reality environment: the end-user, the real and the virtual objects. The second is due to the ever-increasing participation of sensors controlling devices and machines always and everywhere. This paper highlights these synergies and proposes a case proving they crucially empower a Mixed Reality experience. Through this synergy, a novel interaction method for Mixed Reality Environments emerges, the “physical task execution” interaction method. An open-source application prototype was developed to support the case study where we assign a physical task to a virtual man to execute, utilising the Internet of Things in a Mixed Reality Environment.

A user experience analysis for a mobile Mixed Reality application for cultural heritage

AbstractMixed Reality has emerged as a valuable tool for the promotion of cultural heritage. In this context, in particular, the metaphor of virtual portals allows the virtual visit of monuments that are inaccessible or no longer exist in their original form, integrating them into the real environment. This paper presents the development of a Mixed Reality mobile application that proposes a virtual reconstruction of the church of Sant’Elia in Ruggiano, in the southern province of Lecce (Italy). By placing the virtual portal in the same place where the entrance of the church was located, the user can cross this threshold to enter inside and make a virtual journey into the past. The user experience was evaluated by administering a questionnaire to 60 users who tried the application. From the data collected, four user experience factors were identified (interest, focus of attention, presence and usability), which were compared between young and old, male and female users, and between users who had already visited the church in person and all other users. In general, the scores reveal a total independence of the other three factors from usability and a very high level of interest.

A new mixed reality tool for training in minimally invasive robotic-assisted surgery

Robotic-assisted surgery (RAS) is developing an increasing role in surgical practice. Therefore, it is of the utmost importance to introduce this paradigm into surgical training programs. However, the steep learning curve of RAS remains a problem that hinders the development and widespread use of this surgical paradigm. For this reason, it is important to be able to train surgeons in the use of RAS procedures. RAS involves distinctive features that makes its learning different to other minimally invasive surgical procedures. One of these features is that the surgeons operate using a stereoscopic console. Therefore, it is necessary to perform RAS training stereoscopically. This article presents a mixed-reality (MR) tool for the stereoscopic visualization, annotation and collaborative display of RAS surgical procedures. The tool is an MR application because it can display real stereoscopic content and augment it with virtual elements (annotations) properly registered in 3D and tracked over time. This new tool allows the registration of surgical procedures, teachers (experts) and students (trainees), so that the teacher can share a set of videos with their students, annotate them with virtual information and use a shared virtual pointer with the students. The students can visualize the videos within a web environment using their personal mobile phones or a desktop stereo system. The use of the tool has been assessed by a group of 15 surgeons during a robotic-surgery master’s course. The results show that surgeons consider that this tool can be very useful in RAS training.

MIXED REALITY APPLICATION FOR STAIRCASE INSPECTION

Mixed Reality (MR) solutions for the construction industry have been studied, with many bringing in a Building Information Model (BIM) within the mixed reality environment for visualization to assist with inspections. While some studies have been done to determine the efficiency of mapping indoor scenes using mixed-reality headsets, few have conducted research for specific applications that utilize the mapping capabilities of mixed-reality hardware. In the construction industry, building authority standards have to be complied with, which are typically manually inspected. Wearable mixed reality solutions could potentially be used to facilitate such inspections by using native sensors available in the hardware to make sense of the environment, followed by a comparison of the acquired data against a set of authority requirements. The objective of the research is to study the feasibility of utilizing a wearable mixed reality device as a tool for building staircase inspection, where such inspections are usually tedious since it involves manual measurements of several staircase parameters. Using data gathered from sensors on the device, holograms representing staircase features are generated for visualization and computing distance measurements, which are then used for checks against a set of standards. Due to hardware limitations, the application was only able to gather and analyze data for parts of staircases with reasonable accuracy. Nonetheless, the mixed reality application showed that simpler structures with fewer features than staircases can be successfully deployed for measurement-type inspection checks.

CYBERSECURITY RECOMMENDATIONS AND BEST PRACTICES FOR DIGITAL EDUCATION

In light of Covid-19, there has been an increasing trend over the past few years to use e-learning platforms more frequently. Hackers use various methods to play with the human mind of the learner or instructor, using their unexperienced internet habits. The article examines the risks and benefits of using augmented, mixed, and virtual reality applications in e-learning. Good practices for cyber risk assessment and measures to improve cyber security are shown.