Mixed Reality Applications Research Articles

Mixed reality applications in upper extremity surgery: the future is now.

Mixed reality refers to the integration of virtual reality into the real-world environment. This digital content can be interacted with in real time. The emergence of mixed reality technology has been made possible by the introduction of head-mounted displays, which are being utilized across multiple surgical specialties. In upper extremity surgery, mixed reality has widespread applications in trauma, corrective surgery, arthroplasty, arthroscopy, and oncology. Preoperatively, mixed reality allows for complex 3D planning. Intraoperatively, surgeons can access this 3D data in a sterile environment. While in its infant stages, mixed reality is likely to become a powerful tool for intraoperative guidance and navigation. Mixed reality can change the paradigm of communication, as it allows the sharing of visual data from the surgeon's perspective, enabling remote assistance and participation.

Breakthrough and Challenging Application: Mixed Reality-Assisted Intracardiac Surgery

Background: While several studies investigate the utility and clinical value of 3D printing in aiding diagnosis, medical education, preoperative planning, and intraoperative guidance of surgical interventions, there is a scarcity of literature regarding concrete applications of mixed reality in the cardiovascular domain due to its nascent stage of study and expansion. This study goes beyond a mere three-dimensional visualization of the cardiac district, aiming to visualize the intracardiac structures within the scope of preoperative planning for cardiac surgery. Methods: The segmentation of the heart was performed through an open-source and a professional software and by applying different procedures. Each anatomical component of the heart, including the aortic valve, was accurately segmented and a 3D model was built to represent the entire heart. Results: Beyond the three-dimensional visualization of the cardiac region, the intracardiac structures were also segmented. A mixed-reality app was implemented with the possibility of exploding the model, interacting with it, and freely sectioning it with a plane. Conclusions: The proposed segmentation methodology allows a segmentation of the valve and the intracardiac structures. Furthermore, the mixed-reality app has confirmed the potential of this technology in diagnostic and preoperative planning, although some limitations should still be overcome.

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion.

3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Precise Tool to Target Positioning Widgets (TOTTA) in Spatial Environments: A Systematic Review.

TOTTA outlines the spatial position and rotation guidance of a real/virtual tool (TO) towards a real/virtual target (TA), which is a key task in Mixed reality applications. The task error can have critical consequences regarding safety, performance, and quality, such as surgical implantology or industrial maintenance scenarios. The TOTTA problem lacks a dedicated study and it is scattered in different domains with isolated designs. This work contributes to a systematic review of the TOTTA visual widgets, studying 70 unique designs from 24 papers. TOTTA is commonly guided by the visual overlap -an intuitive, pre-attentive "collimation" feedback- of simple shaped widgets: Box, 3D Axes, 3D Model, 2D Crosshair, Globe, Tetrahedron, Line, Plane. Our research discovers that TO and TA are often represented with the same shape. They are distinguished by topological elements (e.g. edges/vertices/faces), colors, transparency levels, and added. shapes, widget quantity, and size. Meanwhile some designs provide continuous "during manipulation feedback" relative to the distance between TO and TA by text, dynamic color, sonification, and amplified graphical visualization. Some approaches trigger discrete "TA reached feedback" such as color alteration, added sound, TA shape change, and added text. We found the lack of golden standards, including in testing procedures, as current ones are limited to partial sets with different and incomparable setups (different target configurations, avatar, background, etc.). We also found a bias in participants: right-handed, young male, non-color impaired.

A Real-Time and Interactive Fluid Modeling System for Mixed Reality.

Within the realm of mixed reality, the capability to dynamically render environmental effects with high realism plays a crucial role in amplifying user engagement and interaction. Fluid dynamics, in particular, stand out as essential elements for crafting immersive virtual settings. This includes the simulation of phenomena like smoke, fire, and clouds, which are instrumental in enriching the virtual experience. This work showcases a cutting-edge system developed to produce dynamic and interactive fluid effects that mirror real captured data in real-time for mixed reality applications. This innovative system seamlessly incorporates fluid reconstruction alongside velocity estimation processes within the Unity engine environment. Our approach leverages a novel physics-based differentiable rendering technique, grounded in the principles of light transport in participating media, to simulate the intricate behaviors of fluid while ensuring high fidelity in visual appearance. To further enhance realism, we have expanded our framework to include the estimation of velocity fields, addressing the critical need for fluid motion simulation. The practical application of these techniques demonstrates the system's capacity to offer a robust platform for fluid modeling in mixed reality environments. Through extensive evaluations, we illustrate the effectiveness of our approach in various scenes, underscoring its potential to transform mixed reality content creation by providing developers with the tools to incorporate highly realistic and interactive fluid seamlessly.

Mixed reality combined with surgical navigation technology assisted in parapharyngeal space tumors surgery: a clinical study

Objective: To assess the feasibility and application of mixed reality combined with surgical navigation technology in parapharyngeal space tumor surgery, and to provide a reference for the development and promotion of this technology. Methods: In this study, retrospective data collection was conducted on 16 patients with parapharyngeal space tumors who were treated at the Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from June 2020 to June 2023. The patient's age was (39.6±17.8) years, with 4 males and 12 females. Mixed reality combined with surgical navigation technology was utilized to assist physicians in the treatment of these patients. Mixed reality combined with surgical navigation technology was used to assist physicians in treatment of these patients. The application steps included acquisition of image data, processing of image data [three-dimensional (3D) reconstruction, image fusion, and virtual surgical design], development of surgical navigation plan, connection of mixed reality and navigation system, automatic registration and intraoperative guidance and validation. In the preoperative plan, landmark points were placed on the virtual tumor and surrounding important structures reconstructed using digital software, serving to guide the localization of crucial anatomical structures. Intraoperative positioning deviation, surgical time, intraoperative blood loss, and postoperative complications were recorded and analyzed to evaluate the clinical application effectiveness of mixed reality combined with surgical navigation technology. Results: With the assistance of mixed reality combined with surgical navigation technology, 16 patients successfully underwent tumor resection. All patients were accurately diagnosed preoperatively by 3D reconstruction and image fusion technology, and a comprehensive preoperative plan was formulated; intraoperatively, mixed reality combined with surgical navigation technology was utilized for the localization of important structures. The average localization deviation of 38 landmark points during the operation were (4.43±1.96) mm, with 62% (26/42) of the points having a deviation of ≥0 and<5 mm. The average duration of the operation was (149.6±53.9) min and the blood loss was 70 (45, 150) ml. The average postoperative follow-up was 16 months, and five patients experienced postoperative complications involving facial paralysis, hoarseness, and choking. Conclusions: Mixed reality combined with surgical navigation technology can achieve the three-dimensional visualization of oral and maxillofacial anatomical structures to achieve precise preoperative diagnosis. During surgery, the technology can real-time display the relationship between soft tissue tumors and the surrounding important anatomical structures, guide surgical operation, and enhance the safety of surgery.

Augmented and virtual reality imaging for collaborative planning of structural cardiovascular interventions: a proof-of-concept and validation study.

Virtual reality (VR) and augmented reality (AR) have led to significant advancements in cardiac preoperative planning, shaping the world in profound ways. A noticeable gap exists in the availability of a comprehensive multi-user, multi-device mixed reality application that can be used in a multidisciplinary team meeting. A multi-user, multi-device mixed reality application was developed, supporting AR and VR implementations. Technical validation involved a standardized testing protocol and comparison of AR and VR measurements regarding absolute error and time. Preclinical validation engaged experts in interventional cardiology, evaluating the clinical applicability prior to clinical validation. Clinical validation included patient-specific measurements for five patients in VR compared with standard computed tomography (CT) for preoperative planning. Questionnaires were used at all stages for subjective evaluation. Technical validation, including 106 size measurements, demonstrated an absolute median error of 0.69mm (0.25 to 1.18mm) compared with ground truth. The time to complete the entire task was on average, with VR measurements being faster than AR ( versus , ). On clinical validation of five preoperative patients, there was no statistically significant difference between paired CT and VR measurements (0.58 [95% CI, to 2.74], ). Questionnaires showcased unanimous agreement on the user-friendly nature, effectiveness, and clinical value. The mixed reality application, validated through technical, preclinical, and clinical assessments, demonstrates precision and user-friendliness. Further research of our application is needed to validate the generalizability and impact on patient outcomes.

Application of Mixed Reality Technologies in Autism Treatment and Neurodiversity Support

Abstract: This comprehensive review explores the application of Mixed Reality (MR) technologies in treating Autism Spectrum Disorder (ASD) and supporting people with neurodiversity. It examines the potential of MR to create immersive, personalized interventions for social skills training, communication enhancement, and behavioral therapy. The paper discusses the technical considerations in implementing MR systems, including hardware requirements, software development, and user interface design tailored for neurodiverse individuals. It also addresses the challenges and future directions in this field, including ethical considerations, the need for clinical validation, and the integration of MR with existing therapeutic approaches. By synthesizing current research and identifying areas for future investigation, this review highlights the transformative potential of MR in revolutionizing ASD therapy and support.

Design of Virtual Reality System for 3D Visualization and Interaction

Nowadays, a virtual environment or metaverse environment is becoming more popular and can be used for various purposes. There are many computer games and more applications for them such as for training, learning and planning. Currently the presentations of 3-Dimensional (3D) models using mixed reality (MR) technology can be used to avoid misinterpretations of oral and 2-Dimensional (2D) model presentations. As an independent concept and MR applications, MR combines the best of virtual reality (VR) and augmented reality (AR). Therefore, this research aimed to present the descriptions of MR systems, which include its devices, applications, literature reviews and create 3D models that were then implemented in Unity 3D using VR applications for 3D architectural visualization. In addition, the assessment results of the VR application showed that the mean score of 2.85 and 3.45 (“neutral”) indicated that the VR application was a new one for the students and they had never used it before, but they certify that learning with VR is not as difficult as with other simulations. The mean scores of 4.55, 4.70, and 4.75 (“strongly agree”) indicated that the use of the VR application’s 3D objects in the scene was beautiful, clear, and realistic. Besides, the student also enjoyed the immersive experience and the learning was very interesting and they felt that they were in the real world.

Educational utilization of virtual and mixed reality

This study investigated the application and acceptance of virtual reality (VR) and mixed reality (MR) technologies in university education. Here, university freshmen were introduced to these technologies and surveyed to assess their perceptions of VR and MR systems when applied in educational settings. The results demonstrate that the participants generally responded positively to both technologies, with VR being rated higher in terms of enjoyment and perceived usefulness. The findings suggest that students are more inclined to use VR in future educational settings. MR also received favorable evaluations; however, it received lower evaluations, particularly in terms of ease of use, which is likely due to the participants’ relative inexperience with the technology. The findings of this study highlight the potential of VR and MR technologies as effective educational tools and emphasize the importance of leveraging their respective advantages.

Design of a Mixed-Reality Application to Reduce Pediatric Medication Errors in Prehospital Emergency Care

Children in prehospital emergency care are particularly vulnerable to medication errors, often with serious consequences. A prior study analyzing prehospital pediatric medication dosing errors, conducted after the implementation of a statewide pediatric drug-dosing reference for emergency medical services (EMS), identified an alarmingly high error rate. This significant finding led to the current study, which aims to develop technological interventions to reduce the frequency of medication errors for children during treatment by EMS. The current study focuses on the design and development of a safety strategy to automate medication administration using mixed-reality technology. Simulations were conducted to inform the design process, focusing on three scenarios: cardiac arrest, seizure, and burns. The design team included medical and engineering researchers, paramedics, and emergency medical technicians from multiple emergency medical service agencies. Root cause analysis (RCA) and failure mode and effects analysis (FMEA) were conducted after the simulations were completed. The RCA and FMEA were used to identify and prioritize failure points, which were then addressed in a mixed-reality solution using Microsoft HoloLens 2 to automate and enhance pediatric medication administration in prehospital emergency care. The resulting application will provide real-time assistance to guide paramedics through the complicated medication dosing and administration process using a detailed step-by-step guide, aiming to decrease medication errors and improve medication dosing accuracy.

Using Mixed Reality for Control and Monitoring of Robot Model Based on Robot Operating System 2

This article presents the design and implementation of an innovative human–machine interface (HMI) in mixed reality for a robot model operating within Robot Operating System 2 (ROS 2). The interface is specifically developed for compatibility with Microsoft HoloLens 2 hardware and leverages the Unity game engine alongside the Mixed Reality Toolkit (MRTK) to create an immersive mixed reality application. The project uses the Turtlebot 3 Burger model robot, simulated within the Gazebo virtual environment, as a representative mechatronic system for demonstration purposes. Communication between the mixed reality application and ROS 2 is facilitated through a publish–subscribe mechanism, utilizing ROS TCP Connector for message serialization between nodes. This interface not only enhances the user experience by allowing for the real-time monitoring and control of the robotic system but also aligns with the principles of Industry 5.0, emphasizing human-centric and inclusive technological advancements. The practical outcomes of this research include a fully functional mixed reality application that integrates seamlessly with ROS 2, showcasing the potential of mixed reality technologies in advancing the field of industrial automation and human–machine interaction.

Mixed Reality in Clinical Settings for Pediatric Patients and Their Families: A Literature Review.

In the post-pandemic context, there has been an increasing demand for technology-based interventions in education and healthcare systems, such as augmented and mixed reality technologies. Despite the promising outcomes of applying mixed reality (MR), there is limited aggregated evidence focusing on child-patient interventions in hospital-based or clinical settings. This literature review aimed to identify and synthesize existing knowledge on MR technologies applied to pediatric patients in healthcare settings. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a comprehensive search of the Scopus and Web of Science databases was conducted to identify articles published in the last 10 years that address the application of augmented and/or MR technologies in pediatric hospital settings or clinical environments to improve patient and family outcomes. A total of 45 articles were identified, and following a rigorous screening and eligibility process, 4 review articles were selected for qualitative synthesis. From these reviews, 10 studies with relevant interventions and measured effects were extracted. The extracted studies were analyzed based on eight key attributes: country of origin, study design, characteristics of the study population, primary clinical setting, type of MR device used, nature of the intervention, variables measured, and significant effects observed in the outcome variables. The analysis revealed diverse approaches across different clinical settings, with a common focus on improving both emotional well-being and learning outcomes in pediatric patients and their families. These findings suggest that MR-based pediatric interventions generally provide children and their parents with positive emotional experiences, enhancing both learning and treatment outcomes. However, the studies reviewed were heterogeneous and varied significantly in terms of clinical settings and MR applications. Future research should focus on developing more controlled study designs that specifically target the pediatric population to strengthen the evidence base for MR interventions in healthcare.

Developing a conceptual framework for usability evaluation of cartographic design principles in Mixed Reality applications

Developing a conceptual framework for usability evaluation of cartographic design principles in Mixed Reality applications

Mixed Reality in Medical Education - Introduction of a Practical Course Module.

With increasing digitalization in healthcare, there is growing potential for the use of Mixed Reality (MR) in medicine. In response to this, we developed a course to provide students with hands-on experience in MR and familiarize them with the technology. Utilizing the HoloLens 2, we designed an application enabling multiple users to interact with virtual objects overlaid on the real world and shared across devices. We used 3D heart anatomy models to let students discuss cardiovascular diseases together. We conducted a preliminary evaluation of our prototype, involving ten participants. The first test indicated a positive reception of the course, with students expressing keen interest in MR. With this work we share our experience to contribute to the field of MR applications.

MIXED-REALITY GUIDANCE FOR PELVIC OSTEOTOMY AND ACETABULAR REALIGNMENT: PROOF OF CONCEPT

Mixed Reality has the potential to improve accuracy and reduce required dissection for the performance of peri-acetabular osteotomy. The current work assesses initial proof of concept of MR guidance for PAO.A PAO planning module, based on preoperative computed tomography (CT) imaging, allows for the planning of PAO cut planes and repositioning of the acetabular fragment. 3D files (holograms) of the cut planes and native and planned acetabulum positions are exported with the associated spatial information. The files are then displayed on mixed reality head mounted device (HoloLens2, Microsoft) following intraoperative registration using an FDA-cleared mixed reality application designed primary for hip arthroplasty (HipInsight). PAO was performed on both sides of a bone model (Pacific Research). The osteotomies and acetabular reposition were performed in accordance with the displayed holograms. Post-op CT imaging was performed for analysis. Cutting plane-accuracy was evaluated using a best-fit plane and 2D angles (°) between the planned and achieved supra (SA)- and retroacetabular (RA) osteotomy and retroacetabular and ischial osteotomies (IO) were measured.To evaluate the accuracy of acetabular reorientation, we digitized the acetabular rim and calculated the acetabular opening plane. Absolute errors of planned and achieved operative inclination and anteversion (°) of the acetabular fragment, as well as 3D lateral-center-edge (LCE) angles were calculated.The mean absolute difference between the planned and performed osteotomy angles was 3 ± 3°.The mean absolute error between planned and achieved operative anteversion and inclination was 1 ± 0° and 0 ± 0° respectively. Mean absolute error between planned and achieved 3D LCE angle was 0.5 ± 0.7°.Mixed-reality guidance for the performance of pelvic osteotomies and acetabular fragment reorientation was feasible and highly accurate. This solution may improve the current standard of care by enabling reliable and precise reproduction of the desired acetabular realignment.

Ultra-low-defect homoepitaxial micro-LEDs with enhanced efficiency and monochromaticity for high-PPI AR/MR displays

The issue of brightness in strong ambient light conditions is one of the critical obstacles restricting the application of augmented reality (AR) and mixed reality (MR). Gallium nitride (GaN)-based micro-LEDs, renowned for their exceptional brightness and stability, are considered the foremost contenders for AR applications. Nevertheless, conventional heteroepitaxial growth micro-LED devices confront formidable challenges, including substantial wavelength shifts and efficiency droop. In this paper, we firstly demonstrated the high-quality homoepitaxial GaN-on-GaN micro-LEDs micro-display, and thoroughly analyzed the possible benefits for free-standing GaN substrate from the material-level characterization to device optoelectronic properties and micro-display application compared with sapphire substrate. The GaN-on-GaN structure exhibits a superior crystal quality with ultra-low threading dislocation densities (TDDs) of ~ 105 cm−2, which is three orders of magnitude lower than that of GaN-on-Sapphire. Through an in-depth size-dependent optoelectronic analysis of blue/green emission GaN-on-GaN/ Sapphire micro-LEDs from 100 × 100 shrink to 3 × 3 μm2, real that a lower forward voltage and series resistance, a consistent emission wavelength (1.21 nm for blue and 4.79 nm for green @ 500 A/cm2), coupled with a notable reduction in efficiency droop ratios (15.6% for blue and 28.5% for green @ 500 A/cm2) and expanded color gamut (103.57% over Rec. 2020) within GaN-on-GaN 10 μm micro-LEDs. Last but not least, the GaN-on-GaN micro-display with 3000 pixels per inch (PPI) showcased enhanced display uniformity and higher luminance in comparison to its GaN-on-Sapphire counterpart, demonstrating significant potentials for high-brightness AR/MR applications under strong ambient light.

Advanced outcomes of mixed reality usage in orthognathic surgery: a systematic review

IntroductionOrthognathic surgery (OGS) is a highly sophisticated surgical technique that aims to repair a variety of skeletal and dental abnormalities, including misaligned jaws and teeth. It requires precise preoperative preparation and advanced surgical skills, which are typically learned through years of practical experience in operating rooms or laboratory-based surgical training facilities utilizing cadavers or models. The traditional physical hands-on method of surgical training is still used at OGS. However, this method requires a longer time of preparation. Currently, mixed reality (MR)—a combination of virtual reality and augmented reality technology—is an innovation of OGS. The present study aimed to present a comprehensive review of studies that assessed the advantages of utilizing mixed reality technology in OGS.MethodsA modified Population, Intervention, Comparison, Outcome strategy was performed using a combination of electronic (PubMed, Cochrane, Embase) and manual searches between 2013 and 2023 exploring mixed reality (MR) technology in OGS in the last 10 years. The inclusion criteria were limited to the patient and study model focusing on the clinical application of MR and the associated field of OGS.ResultThe initial search indicated 1731 studies, of which 17 studies were included for analysis. The main results indicated that the use of MR technology in OGS led to high accuracy and time reduction as primary outcomes and cost-effectiveness and skill improvement as secondary outcomes. The review firmly concluded that MR technology exhibited a positive impact on students, trainees, and oromaxillofacial surgeons. However, due to the heterogeneity of the included studies, meta-analyses could not be performed. Collectively, these findings provide strong evidence for the advantages of MR technology in orthognathic surgery.ConclusionMR technology significantly improves OGS planning efficiency by providing pre-surgical information and serving as an intraoperative navigation tool, reducing surgical time without compromising outcomes. Virtual training using MR technology exerts a positive impact on knowledge and skill improvement for OGS. This innovative technology will revolutionize the healthcare system and enhance patient care.

Application of Mixed/Augmented Reality in Interventional Cardiology.

This review explores the transformative applications of augmented reality (AR) and mixed reality (MR) technologies in interventional cardiology. The integration of these cutting-edge systems offers unprecedented potential to enhance visualization, guidance, and outcomes during complex cardiac interventional procedures. This review examines four key domains: (1) medical AR/MR systems and technological foundations; (2) clinical applications across procedures like TAVI, PCI, and electrophysiology mapping; (3) ongoing technology development and validation efforts; and (4) educational and training applications for fostering essential skills. By providing an in-depth analysis of the benefits, challenges, and future directions, this work elucidates the paradigm shift catalyzed by AR and MR in advancing interventional cardiology practices. Through meticulous exploration of technological, clinical, and educational implications, this review underscores the pivotal role of these innovative technologies in optimizing procedural guidance, improving patient outcomes, and driving innovation in cardiovascular care.

Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: enhancements in surgical training, planning, and navigation.

Recent advances in medical imaging, computer vision, 3-dimensional (3D) modeling, and artificial intelligence (AI) integrated technologies paved the way for generating patient-specific, realistic 3D visualization of pathological anatomy in neurosurgical conditions. Immersive surgical simulations through augmented reality (AR), virtual reality (VR), mixed reality (MxR), extended reality (XR), and 3D printing applications further increased their utilization in current surgical practice and training. This narrative review investigates state-of-the-art studies, the limitations of these technologies, and future directions for them in the field of skull base surgery. We begin with a methodology summary to create accurate 3D models customized for each patient by combining several imaging modalities. Then, we explore how these models are employed in surgical planning simulations and real-time navigation systems in surgical procedures involving the anterior, middle, and posterior cranial skull bases, including endoscopic and open microsurgical operations. We also evaluate their influence on surgical decision-making, performance, and education. Accumulating evidence demonstrates that these technologies can enhance the visibility of the neuroanatomical structures situated at the cranial base and assist surgeons in preoperative planning and intraoperative navigation, thus showing great potential to improve surgical results and reduce complications. Maximum effectiveness can be achieved in approach selection, patient positioning, craniotomy placement, anti-target avoidance, and comprehension of spatial interrelationships of neurovascular structures. Finally, we present the obstacles and possible future paths for the broader implementation of these groundbreaking methods in neurosurgery, highlighting the importance of ongoing technological advancements and interdisciplinary collaboration to improve the accuracy and usefulness of 3D visualization and reality technologies in skull base surgeries.