Mixed Reality Environment Research Articles

A Real-Time and Interactive Fluid Modeling System for Mixed Reality.

Within the realm of mixed reality, the capability to dynamically render environmental effects with high realism plays a crucial role in amplifying user engagement and interaction. Fluid dynamics, in particular, stand out as essential elements for crafting immersive virtual settings. This includes the simulation of phenomena like smoke, fire, and clouds, which are instrumental in enriching the virtual experience. This work showcases a cutting-edge system developed to produce dynamic and interactive fluid effects that mirror real captured data in real-time for mixed reality applications. This innovative system seamlessly incorporates fluid reconstruction alongside velocity estimation processes within the Unity engine environment. Our approach leverages a novel physics-based differentiable rendering technique, grounded in the principles of light transport in participating media, to simulate the intricate behaviors of fluid while ensuring high fidelity in visual appearance. To further enhance realism, we have expanded our framework to include the estimation of velocity fields, addressing the critical need for fluid motion simulation. The practical application of these techniques demonstrates the system's capacity to offer a robust platform for fluid modeling in mixed reality environments. Through extensive evaluations, we illustrate the effectiveness of our approach in various scenes, underscoring its potential to transform mixed reality content creation by providing developers with the tools to incorporate highly realistic and interactive fluid seamlessly.

Collaborative Forensic Autopsy Documentation and Supervised Report Generation Using a Hybrid Mixed-Reality Environment and Generative AI.

Forensic investigation is a complex procedure involving experts working together to establish cause of death and report findings to legal authorities. While new technologies are being developed to provide better post-mortem imaging capabilities-including mixed-reality (MR) tools to support 3D visualisation of such data-these tools do not integrate seamlessly into their existing collaborative workflow and report authoring process, requiring extra steps, e.g. to extract imagery from the MR tool and combine with physical autopsy findings for inclusion in the report. Therefore, in this work we design and evaluate a new forensic autopsy report generation workflow and present a novel documentation system using hybrid mixed-reality approaches to integrate visualisation, voice and hand interaction, as well as collaboration and procedure recording. Our preliminary findings indicate that this approach has the potential to improve data management, aid reviewability, and thus, achieve more robust standards. Further, it potentially streamlines report generation and minimise dependency on external tools and assistance, reducing autopsy time and related costs. This system also offers significant potential for education. A free copy of this paper and all supplemental materials are available at https://osf.io/ygfzx.

Assessment of gamified mixed reality environments for upper limb robotic rehabilitation: pilot study on healthy adults

Exoskeletons for rehabilitation have the potential to aid movement and promote intensive, task-oriented, and personalized motor training. However, robotic-assisted training and user experience could be further improve by including Mixed Reality Environments (MREs) and using a gamification approach. This paper presents the design and evaluation of a MR solution based on Microsoft Hololens 2 for robot-assisted bilateral shoulder training. It is based on gamified MREs designed to guide and encourage bilateral synchronous and asynchronous movements of the upper extremities while a 6 Degrees of Freedom (DOF) exoskeleton (FLOAT) provides motor assistance only to the impared limb. The robotic-assisted bimanual tasks involve the use of rehabilitative tools (such as a dowel rod), as a means to interact with the virtual world and to enable the intact limb guide and control the motions of the disable limb. The gamified MR training solution generates meaningful performance metrics from the kinematic analysis of hands movement, captured by Hololens. The subjective evaluation of the gamified MR solution focused on usability, cognitive load, and user experience. Meanwhile, the objective evaluation encompassed the analysis of the robot-assisted movements with and without gamified MREs, a comparative analysis between metrics obtained from Hololens and Vicon data, and the collection of reference data and trajectories. There were twenty-one healthy adults involved in the evaluation of the system. The results with the gamified MREs highlight excellent system usability, low cognitive load, and high user experience. Additionally, integrating gamified MREs into robot-assisted movements enhances shoulder movements. Data collection with Hololens demonstrated to be reliable and consistent. Furthermore, the normal reference values, paths, and velocity profiles obtained from healthy individuals offer a foundation for assessing the performance of individuals with disabilities. In summary, the introduction of gamified MREs for robot-assisted bilateral shoulder movements marks a significant and promising advancement in rehabilitation technology.

Don’t Block My Stuff: Fostering Personal Object Awareness in Multi-user Mixed Reality Environments

In Mixed Reality (MR), users can collaborate efficiently by creating personalized layouts that incorporate both personal and shared virtual objects. Unlike in the real world, personal objects in MR are only visible to their owner. This makes them susceptible to occlusions from shared objects of other users, who remain unaware of their existence. Thus, achieving unobstructed layouts in collaborative MR settings requires knowledge of where others have placed their personal objects. In this paper, we assessed the effects of three visualizations, and a baseline without any visualization, on occlusions and user perceptions. Our study involved 16 dyads (N=32) who engaged in a series of collaborative sorting tasks. Results indicate that the choice of visualization significantly impacts both occlusion and perception, emphasizing the need for effective visualizations to enhance collaborative MR experiences. We conclude with design recommendations for multi-user MR systems to better accommodate both personal and shared interfaces simultaneously.

Augmented Reality for Exploring Solids: A Study on Improving Pre-Service Teachers

Applications of augmented reality are gradually being integrated into teaching at various levels and types of schools. Augmented reality has the potential to transform the training of future teachers in the context of developing their mathematical knowledge, digital skills, and teaching competencies. The paper focuses on researching the impact of augmented reality on the learning of future primary teachers. The aim was to investigate the impact of AR applications on the development of cognitive and psychomotor skills and on students' motivation to learn. The study had a mixed research design. Using a pedagogical experiment, we investigated the effectiveness of an educational intervention based on the use of applications Augmented Reality in combination with hands on activities in the training of future teachers. The educational intervention was a set of activities that were carried out in a mixed reality environment using the 3D solids construction kit, the Polyedres augmentes and GeoGebra applications. The content of the educational intervention was the exploration of significant elements and properties of solids. We tested students' knowledge and abilities in the following areas: identification of simple solids from a picture; identification of significant elements of solids (vertices, edges and faces); identification of the shapes of faces of solids; nets of solids; simple metric tasks with solids. The study was conducted in 2024 with 152 primary education students as part of geometry courses. In line with the results of previous research in this area, we found that AR has the potential to improve understanding of geometric concepts, increase interactivity and student motivation. Augmented reality opens up new possibilities for exploring solids and their properties, as it allows for transfer between planar and spatial representation of solids and helps to develop spatial imagination.

Enhancing Cognitive Fit: Exploring the Potential of Mixed Reality for Developing Mental Rotation Skills

Mixed Reality is a promising venue for spatial ability training, allowing participants to engage in problem-solving through gesture-based interactions with holographic objects. 3-D measurement of learning in Mixed Reality environments may result in a better cognitive fit than 2-D. This mixed-method study measures and enhances mental rotation ability and investigates user experience in Mixed Reality. To assess mental rotation ability, we adapted the Purdue Spatial Visualization Test:Rotation for Microsoft HoloLens 2, creating a measurement tool for our Mixed Reality-based training program, Holomental. Comparing 2-D and 3-D tests, we explored how stimulus dimensionality influences accuracy and cognitive load. Our findings indicate that Holomental enhances mental rotation performance, both in 2-D and 3-D. Cognitive load in the 3-D test was lower than in the 2-D test. Semi-structured interviews revealed participants’ appreciation for the representational and interactional affordances of the training environment. This study provides valuable insights into the effectiveness of 3-D spatial training, with implications and suggestions for future design considerations.

Audio’s Impact on Immersive Experience in Extended Reality and Digital Games: A Systematic Review

Extended reality and digital games strive to deliver a high level of “immersion,” a complex phenomenon influenced by both perceptual and psychological factors. Audio plays a crucial role in shaping immersive experiences, yet there is no clear consensus on its impact or on the best methods to evaluate it. This paper presents a systematic literature review spanning two decades, outlining the methods and findings related to how audio influences immersion in extended reality and digital games. It reveals a strong preference for experiments using virtual reality headsets and headphones and notes a gap in research on augmented and mixed reality environments. Moreover, it underscores the need for audio-specific metrics to better assess the ways that audio variables impact immersion. The findings demonstrate that audio elements like spatial fidelity, music, and the integration of sound in multimodal environments generally contribute to the immersive experience but also highlight a threshold beyond which further enhancements may not perceptibly improve the experience. It emphasizes the need for realtime, objective measures of immersion as well as the consideration of diverse methodological approaches to deepen the understanding of audio’s role in immersive technologies.

Initial framework design of a digital twin mixed-reality-application on human-robot bi-directional collaboration for forming double curvature plate

This paper explores and identifies commonalities among recent Digital Twin (DT) technology definitions. additionally, it introduces a framework for a digital twin of Human-Robot Collaboration (HRC) focusing on English wheel manufacturing as its foundational context. To achieve this, two 3D simulators, namely Grasshopper and Rhino, were employed to faithfully replicate the English wheel process digitally. Subsequently, optimizing these simulators’ performance paved the way for a comprehensive visualization of the English wheel using the Microsoft HoloLens. This visualization integrates digital and physical interactions, resulting in a mixed reality environment. The ultimate goal of this research is to enable a bi-directional collaboration within the digital representation of the English wheel, bridging the gap between the virtual and physical realms.

Enhancing surgical occlusion setting in orthognathic surgery planning using mixed reality technology: a comparative study

ObjectivesOrthognathic surgery necessitates precise occlusal alignment during surgical planning, traditionally achieved through manual alignment of physical dental models as the recognized gold standard. This study aims to evaluate the efficacy of mixed reality technology in enhancing surgical occlusion setting compared to traditional physical alignment and an established virtual method, addressing the research question: Can mixed reality technology improve the accuracy and efficiency of occlusion setting in orthognathic surgery planning?Materials & methodsThis experimental study compared the surgical occlusion settings of 30 orthognathic cases using three methods: a new virtual method with mixed reality technology, the traditional gold standard of physical alignment, and an established virtual occlusion method using the IPS Case Designer (KLS Martin SE & Co. KG, Tuttlingen, Germany).ResultsResults indicated that surgical occlusions set with mixed reality technology were comparable to the conventional method in terms of maxillary movement and occlusal relationship. Differences observed were within the inter-observer variability of the gold standard. Both virtual methods tended to position the maxilla more anteriorly, resulting in fewer occlusal contacts. However, virtual occlusion demonstrated clinical applicability, achieving an average of 11 occlusal contacts with a bilaterally symmetrical distribution along the dental arch.ConclusionsThe mixed reality environment provides an intuitive and flexible experience for setting surgical occlusion, eliminating the need for costly 3D-printed physical models or the automatic calculations required by other virtual occlusion methods, thereby offering maximum freedom.Clinical relevanceAs a novel form of virtual occlusion, it presents a comprehensive tool that contributes to a timely and cost-effective full digital workflow of orthognathic surgery planning.

Evaluating Usability and UX in Touchable Holographic Solutions: A Validation Study of the UUXE-ToH Questionnaire

Touchable Holographic Solutions (THS) are an application type in Augmented and Mixed Reality (AR/MR) environments. Evaluating THS requires considering Usability and User eXperience (UX), particularly Immersion and Presence, often overlooked by current methods. A systematic mapping study and exploratory search revealed gaps, such as the lack of a unified instrument for THS usability and UX evaluation, methods that address THS-specific aspects, and challenges in adapting general evaluation technologies. To bridge these gaps, we present the Usability and User Experience Evaluation in Touchable Holograms (UUXE-ToH) questionnaire, with 67 Likert scale items and three open-ended questions. A validation study with 13 experts provided quantitative data and qualitative feedback. Results confirm the UUXE-ToH’s efficacy and offer insights for improvement. This research contributes to Human-Computer Interaction (HCI) by introducing a validated tool for evaluating THS, integrating hedonic and pragmatic aspects, and supporting early problem identification in development, accessible to professionals with diverse UX expertise.

Sense of embodiment with synchronized avatar during walking in mixed reality.

Gait guidance systems that synchronize the gait rhythm with an avatar in a mixed reality (MR) environment are attracting attention owing to their rehabilitation applications. More effective gait guidance can be achieved by changing body sensations for the sense of embodiment (SoE), which refers to the feeling of owning, controlling, and being inside a body in MR. This study investigated full-body synchronous motion between a human and a virtual avatar to enhance the SoE in walking with actual position changes in the real world. The full-body motion and gait rhythm were measured using body-worn inertial measurement units and a visual avatar was provided through a transparent head-mounted display. The results showed that the SoE of the participants was enhanced under higher synchronization conditions. In addition, questionnaire results showed that the SoE in the synchronous condition was significantly higher than that in the asynchronous condition, and the SoE in the self-avatar condition was significantly higher than that in the other-avatar condition. This indicates that a higher synchronization level with the appearance of an avatar leads to a stronger SoE in the human perception mechanism, which is important for potential application in medical or other fields.

Investigating the Correlation Between Presence and Reaction Time in Mixed Reality.

Measuring presence is critical to improving user involvement and performance in Mixed Reality (MR). Presence, a crucial aspect of MR, is traditionally gauged using subjective questionnaires, leading to a lack of time-varying responses and susceptibility to user bias. Inspired by the existing literature on the relationship between presence and human performance, the proposed methodology systematically measures a user's reaction time to a visual stimulus as they interact within a manipulated MR environment. We explore the user reaction time as a quantity that can be easily measured using the systemic tools available in modern MR devices. We conducted an exploratory study (N = 40) with two experiments designed to alter the users' sense of presence by manipulating place illusion and plausibility illusion. We found a significant correlation between presence scores and reaction times with a correlation coefficient -0.65, suggesting that users with a higher sense of presence responded more swiftly to stimuli. We develop a model that estimates a user's presence level using the reaction time values with high accuracy of up to 80%. While our study suggests that reaction time can be used as a measure of presence, further investigation is needed to improve the accuracy of the model.

Smart and user-centric manufacturing information recommendation using multimodal learning to support human-robot collaboration in mixed reality environments

The future manufacturing system must be capable of supporting customized mass production while reducing cost and must be flexible enough to accommodate market demands. Additionally, workers must possess the knowledge and skills to adapt to the evolving manufacturing environment. Previous studies have been conducted to provide customized manufacturing information to the worker. However, most have not considered the worker's situation or region of interest (ROI), so they had difficulty providing information tailored to the worker. Thus, a manufacturing information recommendation system should utilize not only manufacturing data but also the worker's situational information and intent to assist the worker in adjusting to the evolving working environment. This study presents a smart and user-centric manufacturing information recommendation system that harnesses the vision and text dual encoder-based multimodal deep learning model to offer the most relevant information based on the worker's vision and query, which can support human-robot collaboration (HRC) in a mixed reality (MR) environment. The proposed recommendation model can assist the worker by analyzing the manufacturing environment image acquired from smart glasses, the worker's specific question, and the related manufacturing document. By establishing correlations between the MR-based visual information and the worker's query using the multimodal deep learning model, the proposed approach identifies the most suitable information to be recommended. Furthermore, the recommended information can be visualized through MR smart glasses to support HRC. For quantitative and qualitative evaluation, we compared the proposed model with existing vision-text dual models, and the results demonstrated that the proposed approach outperformed previous studies. Thus, the proposed approach has the potential to assist workers more effectively in MR-based manufacturing environments, enhancing their overall productivity and adaptability.

Investigating the effect of key size, typing angle, and typing technique of virtual keyboard on typing productivity, biomechanics, and usability in a mixed reality environment

This study investigates the effects of key sizes, typing angles, and typing techniques on typing productivity, biomechanics (muscle activity), and subjective experience with a mixed reality keyboard. The findings suggest that using smaller key sizes, such as 16 mm, may not be suitable due to slower typing speed, lower accuracy, lower user experience, higher muscle activity, and higher motion sickness. Typing with both index fingers results in the highest typing speed while using only a single index finger provides higher accuracy. Placing the keyboard at eye height leads to the highest typing speed, as participants can easily view the keys and the virtual environment simultaneously. However, typing accuracy is not affected by typing angle and typing technique. Implementing these findings on the virtual keyboard design could potentially benefit workers’ productivity and decrease errors in the mixed reality environment.

The Potential of Using Interactive Storytelling in a Mixed Reality Environment in Teaching Kazakh Sign Language

The Potential of Using Interactive Storytelling in a Mixed Reality Environment in Teaching Kazakh Sign Language

The Reconstructive Metaverse - Collaboration in Real-Time Shared Mixed Reality Environments for Microsurgical Reconstruction.

Plastic surgeons routinely use 3D-models in their clinical practice, from 3D-photography and surface imaging to 3D-segmentations from radiological scans. However, these models continue to be viewed on flattened 2D screens that do not enable an intuitive understanding of 3D-relationships and cause challenges regarding collaboration with colleagues. The Metaverse has been proposed as a new age of applications building on modern Mixed Reality headset technology that allows remote collaboration on virtual 3D-models in a shared physical-virtual space in real-time. We demonstrate the first use of the Metaverse in the context of reconstructive surgery, focusing on preoperative planning discussions and trainee education. Using a HoloLens headset with the Microsoft Mesh application, we performed planning sessions for 4 DIEP-flaps in our reconstructive metaverse on virtual patient-models segmented from routine CT angiography. In these sessions, surgeons discuss perforator anatomy and perforator selection strategies whilst comprehensively assessing the respective models. We demonstrate the workflow for a one-on-one interaction between an attending surgeon and a trainee in a video featuring both viewpoints as seen through the headset. We believe the Metaverse will provide novel opportunities to use the 3D-models that are already created in everyday plastic surgery practice in a more collaborative, immersive, accessible, and educational manner.

Design and development of a mixed reality application for aphasia rehabilitation: The ICMR-MiRAR project

BACKGROUND: Aphasia is an impaired ability to use language for communication after a brain damage. The primary means of intervention for aphasia – Speech-Language therapy (SLT) – usually involves didactic interaction between the Speech-Language therapist and the client, often without regard to the real-life environments in which the communication occurs. The provision of SLT in natural environments is beyond the scope of the conventional, clinic-based intervention setups. Using the technological advances, the Mixed Reality in Aphasia Rehabilitation (MiRAR) aims to make persons with aphasia (PwA) use their language in an ecologically valid and meaningful manner in natural communication contexts. AIM: This report aims to delineate the design and development of a Mixed Reality environment (MR: i.e., augmented + virtual realities (i.e., AR + VR)) to provide social communicative intervention for PwA. METHODS: We describe the concept and provide the details of the development and deployment of a communication-based mixed reality application for PwA in the Indian context. For this purpose, we generated 20 distinct communication scenarios and their scripts. These scenarios were implemented into the Mixed Reality environment with the help of a hired technical team. RESULTS: The 20 scenarios were successfully developed and deployed into the Mixed Reality environment for the purpose of communication intervention for PwA. The program consists of a web-based admin panel (for SLPs) and a Mixed Reality application (for the PwA). CONCLUSIONS: The MiRAR program is expected to foster the delivery of speech-language therapy in a meaningful, controlled and simulated environments by the SLPs, thus alleviating the practical restraints of conventional clinical setups. The clinical trial of this intervention program is planned in the next phase of this ongoing project.

4D thermo-reflectography of cultural heritage. The Codex 4D project: From data acquisition to the implementation in innovative virtual and mixed reality platforms

In an effort to try and improve the dissemination of cultural heritage, in this paper a novel method for the four-dimensional (4D) digitization of artefacts, based on images recorded in the mid-wave infrared spectral range, is presented. Such a method is here applied to ancient manuscripts in the frame of the “Codex 4D: journey in four dimensions into the manuscript” project. In the proposed approach, the three-dimensional geometry of the manuscript is reconstructed by processing the reflectographic images, obtained from different points of view with respect to the artefact, through Structure from Motion techniques. Thermograms obtained by means of Pulsed Thermography are also recorded since they provide a depth-resolved characterization of the artefact that is also integrated into the digital reconstruction along the orthogonal direction to the surface, hereafter referred to as the fourth dimension. The results gathered from humanities and scientific studies, are also mapped onto the 4D model in the form of interactive semantic annotations. The goal of such a reconstruction is to allow users to browse the subsurface elements into the 4D model, thus facilitating the study and the exploration of the manuscripts through the inclusion of information about their literary and art-historical contents, materials, execution techniques, and state of conservation. In addition, virtual and mixed reality environments have been developed for different kind of audience such as expert users and museum public. Despite the experimentation presented here was carried out exclusively on manuscripts, in our opinion the methodology can be successfully applied to other types of artefacts.

Turning the operating room into a mixed-reality environment: a prospective clinical investigation for cerebral aneurysm clipping.

The overall aim of this study was to demonstrate the potential benefit of a novel mixed-reality-head-mounted display (MR-HMD) on the spatial orientation of surgeons. In a prospective clinical investigation, the authors applied for the first time a new multicamera navigation technology in an operating room setting that allowed them to directly compare MR-HMD navigation to standard monitor navigation. In the study, which included 14 patients with nonruptured middle cerebral artery aneurysms, the authors investigated how intuitively and effectively surgical instruments could be guided in 5 different visual navigation conditions. The authors demonstrate that multicamera tracking can be reliably integrated in a clinical setting (usability score 1.12 ± 0.31). Moreover, the technology captures large volumes of the operating room, allowing the team to track and integrate different devices and instruments, including MR-HMDs. Directly comparing mixed-reality navigation to standard monitor navigation revealed a significantly improved intuition in mixed reality, leading to navigation times that were twice as fast (2.1×, p ≤ 0.01). Despite the enhanced speed, the same targeting accuracy (approximately 2.5 mm, freehand tool use) in comparison to monitor navigation could be observed. Intraoperative planning strategies with mixed reality clearly outperformed classic preoperative planning: surgeons scored the mixed-reality plan as the best trajectory in 63% of the cases (chance level 33%). The incorporation of mixed reality in neurosurgical operations marks a significant advancement in the field. The use of mixed reality in brain surgery enhances the spatial awareness of surgeons, enabling more instinctive and precise surgical interventions. This technological integration promises to refine the execution of complex procedures without compromising accuracy.

Immersive Cross-Cultural Journeys: Enhancing Museum Experiences Through Mixed Reality

Mixed reality technology is employed to develop immersive experiences, enhancing visitor engagement within museum environments. There are challenges in interface design and sensory accuracy for such mixed-reality applications. This study explored integration of mixed reality technology in museums, emphasizing the enhancement of cross-cultural understanding and visitor engagement. Using advanced three-dimensional printing, significant artifacts from various cultures, including a notable Korean artifact from the Goryeo period, were replicated and digitized. These artifacts were the core of a mixed reality environment that was created using Unity and the MR Software Development Kit and experienced through a HoloLens 2 Head-Mounted Display. This immersive mixed reality experience, comprising three interactive segments, incorporated eye- and hand-tracking technologies to enable deeper engagement with cultural artifacts. The impact of mixed reality on visitor engagement and cross-cultural comprehension was evaluated, revealing a significant increase in visitor involvement and appreciation of cultural diversity. Notably, this study contributes to the evolving narrative of digital cultural heritage by highlighting the role of mixed reality in fostering a cross-cultural understanding and shaping future museum experiences.