Mixed Reality Research Articles

Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological Features

Mixed reality enables users to immerse themselves in high-workload interaction spaces like office work scenarios. We envision physiologically adaptive systems that can move users into different mixed reality manifestations, to improve their focus on the primary task. However, it is unclear which manifestation is most conducive for high productivity and engagement. In this work, we evaluate whether physiological indicators for engagement can be discriminated for different manifestations. For this, we engaged participants in a typing task in three different mixed reality manifestations (augmented reality, augmented virtuality, virtual reality) and monitored physiological correlates (EEG, ECG, and eye tracking) of users' engagement and workload. We found that users achieved best typing performances in augmented reality and augmented virtuality. At the same time, physiological engagement peaked in augmented virtuality, while workload decreased. We conclude that augmented virtuality strikes a good balance between the different manifestations, as it facilitates displaying the physical keyboard for improved typing performance and, at the same time, allows one to block out the real world, removing many real-world distractors.

Augmented Reality for Exploring Solids: A Study on Improving Pre-Service Teachers

Applications of augmented reality are gradually being integrated into teaching at various levels and types of schools. Augmented reality has the potential to transform the training of future teachers in the context of developing their mathematical knowledge, digital skills, and teaching competencies. The paper focuses on researching the impact of augmented reality on the learning of future primary teachers. The aim was to investigate the impact of AR applications on the development of cognitive and psychomotor skills and on students' motivation to learn. The study had a mixed research design. Using a pedagogical experiment, we investigated the effectiveness of an educational intervention based on the use of applications Augmented Reality in combination with hands on activities in the training of future teachers. The educational intervention was a set of activities that were carried out in a mixed reality environment using the 3D solids construction kit, the Polyedres augmentes and GeoGebra applications. The content of the educational intervention was the exploration of significant elements and properties of solids. We tested students' knowledge and abilities in the following areas: identification of simple solids from a picture; identification of significant elements of solids (vertices, edges and faces); identification of the shapes of faces of solids; nets of solids; simple metric tasks with solids. The study was conducted in 2024 with 152 primary education students as part of geometry courses. In line with the results of previous research in this area, we found that AR has the potential to improve understanding of geometric concepts, increase interactivity and student motivation. Augmented reality opens up new possibilities for exploring solids and their properties, as it allows for transfer between planar and spatial representation of solids and helps to develop spatial imagination.

How Video Passthrough Headsets Influence Perception of Self and Others.

With the increasing adoption of mixed reality (MR) headsets with video passthrough functionality, concerns over perceptual and social effects have surfaced. Building on prior qualitative findings,1 this study quantitatively investigates the impact of video passthrough on users. Forty participants completed a body transfer task twice, once while wearing a headset in video passthrough and once without a headset. Using video passthrough induced simulator sickness, created social absence (another person in the physical room feels less present), altered self-reported body schema, and distorted distance perception. On the other hand, compared with past research that showed perceptual aftereffects from video passthrough, the current study found none. We discuss the broader implications for the widespread adoption of MR headsets and their impact on theories surrounding presence and body transfer.

Application value of surgical navigation system based on deep learning and mixed reality for guiding puncture in percutaneous nephrolithotomy: a retrospective study.

This study was conducted to investigate the clinical value of a navigation system based on deep learning and mixed reality for the treatment of kidney stones with percutaneous nephrolithotomy (PNL), and to improve its theoretical basis for the treatment of kidney stones. The data of 136 patients with kidney stones from October 2021 to December 2023 were retrospectively analyzed. All patients underwent PNL, and were categorized into a control group (Group 1) and a surgical navigation group (Group 2) according to puncture positioning method. Preoperative computed tomography (CT) was performed in both groups. In group 1, procedures were performed under standard ultrasound guidance. PNL was performed with navigation system fused with ultrasound to guide percutaneous puncture in group 2. The baseline information and procedural characteristics of both groups were compared. PNL was successfully performed in both groups. No significant difference was found in the baseline date between the two groups. In group 2, real-time ultrasound images could be accurately matched with CT images with the aid of navigation system. The success rate of single puncture, puncture time, and decrease in hemoglobin were significantly improved in group 2 compared to group 1. (p < 0.05). The application of navigation system based on deep learning and mixed reality in PNL for kidney stones allows for real-time intraoperative navigation, with acceptable accuracy and safety. Most importantly, this technique is easily mastered, particularly by novice surgeons in the field of PNL.

Mixed reality-based coordinate system for enhancing survey knowledge acquisition: a study on allocentric and egocentric perspectives

ABSTRACT Spatial knowledge, especially survey knowledge, plays a pivotal role in people’s daily spatial tasks. However, the inherently limited visual perception and integration of spatial information often leads to inaccuracies in survey knowledge, with far-reaching consequences that affect performance, emotion, and safety in completing spatial tasks. In this study, we developed a mixed reality (MR)-based coordinate system and conducted a comparative user experiment involving 56 participants to explore whether perceiving metric and global spatial references during navigation may enhance the accuracy of user-acquired spatial knowledge. The experiment encompassed a spatial learning task where participants navigated and learned the selected indoor environment, followed by a desktop-based landmark recalling task to evaluate the accuracy of participants’ acquired survey knowledge. The results revealed the following insights: (1) the use of MR-based coordinate system significantly improved egocentric and allocentric spatial relations in participants’ survey knowledge; (2) participants’ errors in spatial relations among landmarks accumulate over distance, whereas the MR-based coordinate system helps maintain the errors at a consistent level; (3) no significant correlations were found between errors in participants’ survey knowledge and environmental familiarity, suggesting that the aiding effect of the MR-based coordinate system was independent of users’ prior knowledge of the environment; (4) participants reported increased mental demand using the MR-based coordinate system, but this did not result in diminished spatial learning performance or additional time pressure and learning efforts. These empirical results contribute to the future study of spatial knowledge acquisition and the design of spatial learning interfaces in mixed reality.

Enhancing Cognitive Fit: Exploring the Potential of Mixed Reality for Developing Mental Rotation Skills

Mixed Reality is a promising venue for spatial ability training, allowing participants to engage in problem-solving through gesture-based interactions with holographic objects. 3-D measurement of learning in Mixed Reality environments may result in a better cognitive fit than 2-D. This mixed-method study measures and enhances mental rotation ability and investigates user experience in Mixed Reality. To assess mental rotation ability, we adapted the Purdue Spatial Visualization Test:Rotation for Microsoft HoloLens 2, creating a measurement tool for our Mixed Reality-based training program, Holomental. Comparing 2-D and 3-D tests, we explored how stimulus dimensionality influences accuracy and cognitive load. Our findings indicate that Holomental enhances mental rotation performance, both in 2-D and 3-D. Cognitive load in the 3-D test was lower than in the 2-D test. Semi-structured interviews revealed participants’ appreciation for the representational and interactional affordances of the training environment. This study provides valuable insights into the effectiveness of 3-D spatial training, with implications and suggestions for future design considerations.

The Integration of Mixed Reality Simulation into Reading Literacy Modules

The Integration of Mixed Reality Simulation into Reading Literacy Modules

Mixed Reality-Based Smart Occupational Therapy Personalized Protocol for Cerebellar Ataxic Patients.

Occupational therapy (OT) is an essential component of patient care, and it is especially beneficial if focused on meaningful activities. For ataxic patients, traditional procedures are currently the most efficient, although without specific guidelines and suggestions for virtual reality integration. In this context, this study proposes Hybrid Smart Rehabilitation (HSR) based on mixed reality (MR) as an aid in overcoming limitations of the traditional OT procedures. MR-HSR is designed specifically for ataxic patients and developed in Unity with the Holographic Remoting setting for run-time intervention on the scene. The subject reaches a book and grabs it with their hand inside a holographic guide with audio-visive feedback. Hand trajectories acquired from eight ataxic patients and eight healthy subjects were compared and new variables were analyzed to evaluate the performance. The Trust in Automation questionnaire was submitted to assess the opinion of the patients. Patients confirmed their trust in the developer and in the improvement that this system can bring to their rehabilitation. The "total time" and "sway area" of the trajectory were statistically significant and, together with the deviation of the trajectory from the main axis of the guide, although not statistically significant, made it possible to build a classifier. The patient-specific MR-HSR can be considered as an integrative tool for assessing the subject's condition by analyzing new quantitative variables which, if matched to the Scale for the Assessment and Rating of Ataxia (SARA), could be the basis of a new index to assess the progressiveness of ataxia.

Audio’s Impact on Immersive Experience in Extended Reality and Digital Games: A Systematic Review

Extended reality and digital games strive to deliver a high level of “immersion,” a complex phenomenon influenced by both perceptual and psychological factors. Audio plays a crucial role in shaping immersive experiences, yet there is no clear consensus on its impact or on the best methods to evaluate it. This paper presents a systematic literature review spanning two decades, outlining the methods and findings related to how audio influences immersion in extended reality and digital games. It reveals a strong preference for experiments using virtual reality headsets and headphones and notes a gap in research on augmented and mixed reality environments. Moreover, it underscores the need for audio-specific metrics to better assess the ways that audio variables impact immersion. The findings demonstrate that audio elements like spatial fidelity, music, and the integration of sound in multimodal environments generally contribute to the immersive experience but also highlight a threshold beyond which further enhancements may not perceptibly improve the experience. It emphasizes the need for realtime, objective measures of immersion as well as the consideration of diverse methodological approaches to deepen the understanding of audio’s role in immersive technologies.

Proceedings of a membrane update symposium: advancements, scientific insights, and future trends for dialysis membranes for enhanced clinical outcomes in end stage kidney disease patients.

From September 6 - 8 2022, the Life/2022 Membrane Symposium was held in Frankfurt, Germany, and transmitted live to a worldwide internet audience. The event was part of the Life/Nephrology Campus initiative, a continuous educational platform for the nephrology community to expand knowledge and share expertise on contemporary topics in chronic kidney disease. We describe recent questions and advances in the field, and we underline challenges in the care of dialysis patients and opportunities for integration of new findings into clinical practice to improve patient outcomes in end stage kidney disease patients. Most patients with kidney failure are on maintenance hemodialysis (MHD). The scientific program of the symposium was developed around topics about the role, functional determinants, technical aspects, limitations, and clinical implications of membranes presently in use. International experts with clinical or technical expertise as well as scientific recognition within the nephrology community were asked to prepare their presentations based on their own experiences, perceptions, opinions, and sources of information. The symposium devoted a major portion to discussing novel approaches for improving membranes and treatment quality, including updates on innovative concepts that may could potentially transform the landscape of kidney replacement therapy for chronic kidney disease patients in the future. The intent was to provide insights into current attention points for healthcare professionals new to the field of MHD, and to test a unique forum for continuing medical education integrating physician and patient experiences to promote changes in clinical practice. Furthermore, the symposium premiered a specifically developed mixed reality holographic 3D model to demonstrate recent dialyzer innovation diminishing protein fouling on membrane surfaces. As a continuous online educational platform for scientific exchange, this Life/2022 event provided online learning opportunities with on-demand content, with all symposium lectures freely available on nephrologycampus.com.

Participatory Design of an Embodied Mixed Reality Experience Aimed to Assessing Individual and Collaborative Behaviors of Children

This study is part of a broader project exploring the integration of Mixed Reality (MR) in fostering prosocial behaviors among children, particularly focusing on interpersonal synchrony (IPS) and its impact on collaboration. IPS, the coordination of rhythmic movements among individuals, has been associated with improved group cohesion and prosocial behaviors, such as collaboration. Traditional assessments of prosociality typically involve structured play activities for dyads, making them difficult to adapt to other settings. To address these limitations, this research aims to design a flexible embodied MR experience that allows children to choose between collaborative or individual engagement, reflecting the natural dynamics of children's play interactions. For this purpose, we conducted an embodied participatory design (PD) and usability study with 50 children aged 8-10 years. We provide a detailed account on how the PD process was orchestrated and an evaluation of user-system interaction, children's perceptions of play modes, and experience satisfaction. The results show how the MR experience, codesigned with children, supports a free play environment, allowing children to choose their preferred play mode. This study highlights the potential of MR in psychological research and child development, providing a new avenue for assessing individual and collaborative behaviors in larger groups of children.

Advancing Mixed Reality Game Development: An Evaluation of a Visual Game Analytics Tool in Action-Adventure and FPS Genres

In response to the unique challenges of Mixed Reality (MR) game development, we developed GAMR , an analytics tool specifically designed for MR games. GAMR aims to assist developers in identifying and resolving gameplay issues effectively. It features reconstructed gameplay sessions, heatmaps for data visualization, a comprehensive annotation system, and advanced tracking for hands, camera, input, and audio, providing in-depth insights for nuanced game analysis. To evaluate GAMR 's effectiveness, we conducted an experimental study with game development students across two game genres: action-adventure and first-person shooter (FPS). The participants used GAMR and provided feedback on its utility. The results showed a significant positive impact of GAMR in both genres, particularly in action-adventure games. This study demonstrates GAMR 's effectiveness in MR game development and suggests its potential to influence future MR game analytics, addressing the specific needs of developers in this evolving area.

Human-Centric Virtual Museum: Redefining the Museum Experience Through Immersive and Interactive Environments

Virtual Museums present a novel approach for audiences to experience cultural content. This paper proposes two new concepts of virtual museums: Human-Centric Virtual Museum, which emphasizes audience interaction experiences, and Content-Centric Virtual Museum, which focuses more on the digitized display of real museum collections and content. With the widespread application of extended reality (XR) technology in this field, people are increasingly inclined towards experiencing highly interactive Human-Centric Virtual Museums. However, evaluating these from a user cognition perspective warrants further investigation. To address this, we design and implement three fundamental forms: Mobile Application (MA), Virtual Reality (VR), and Mixed Reality (MR), and evaluate comprehensive user experiences, including satisfaction, cognitive load, emotional involvement, presence, interactivity, degree of participation and potential negative impact. The experimental results show that in terms of satisfaction and cognitive load, users generally find the MR mode easier to understand and operate. Regarding emotional involvement, audiences in the VR and MR modes are more likely to focus on the current content. Concerning presence, the MR mode enables audiences to perceive the real spatial location, thereby increasing their immersion. Furthermore, compared to the MA mode, both MR and VR modes exhibit stronger interactivity, providing audiences with a more immersive exhibition experience. However, the VR mode may also cause users to feel discomfort during use.

Systematic Review of Adapting Immersive Technology in Enhancing Teaching and Learning for Students in Higher Education

The integration of immersive technology into educational settings has garnered substantial interest as a potential means to improve teaching and learning (TnL)experiences. The dynamic landscape of education demands innovative approaches to engage, motivate, and empower both educators and learners. Immersive technology, including Virtual Reality (VR), Augmented Reality (AR), as well as Mixed Reality (MR), provides uncommon probabilities to transcend traditional classroom boundaries and provide interactive and immersive learning experiences. However, to harness the full potential of immersive technology, a clear understanding of the conceptual frameworks guiding its implementation is essential. This study utilizes the PRISMA approach. Furthermore, the systematic search of academic databases using the Web of Science (WoS) as well as the Scopus database resulted in an extensive selection of studies, reviews and articles related to immersive learning. Following the implementation of an advanced search approach utilizing keywords (1. Immersive learning, 2. AR, as well as 3. Technology Impact on TnL), the systematic review process yielded a diverse array of articles. These articles were identified and included in the review due to their exploration of the conceptual frameworks that underlie the integration of immersive technology in the field of education. Additionally, the review reveals common themes and emerging trends within the field, shedding light on the diverse approaches to leveraging immersive technology for TnL. The final finding data is (n: 29), which reviews identified key themes. Expect validation to be divided into three themes, which are (1) Immersive learning, (2) AR, and (3) Technology Impact on TnL. Meanwhile, the findings emphasize the importance of well-defined theoretical foundations and best practices when implementing immersive technology for educational purposes. The review underscores the importance of ongoing research as well as innovation in the domain of immersive technology to fully exploit its potential for enhancing TnL experiences. In essence, this systematic review adds to the continued conversation about the efficient utilization of immersive technology in education and paves the way for further exploration and advancement in this field.

Augmented and virtual reality imaging for collaborative planning of structural cardiovascular interventions: a proof-of-concept and validation study.

Virtual reality (VR) and augmented reality (AR) have led to significant advancements in cardiac preoperative planning, shaping the world in profound ways. A noticeable gap exists in the availability of a comprehensive multi-user, multi-device mixed reality application that can be used in a multidisciplinary team meeting. A multi-user, multi-device mixed reality application was developed, supporting AR and VR implementations. Technical validation involved a standardized testing protocol and comparison of AR and VR measurements regarding absolute error and time. Preclinical validation engaged experts in interventional cardiology, evaluating the clinical applicability prior to clinical validation. Clinical validation included patient-specific measurements for five patients in VR compared with standard computed tomography (CT) for preoperative planning. Questionnaires were used at all stages for subjective evaluation. Technical validation, including 106 size measurements, demonstrated an absolute median error of 0.69mm (0.25 to 1.18mm) compared with ground truth. The time to complete the entire task was on average, with VR measurements being faster than AR ( versus , ). On clinical validation of five preoperative patients, there was no statistically significant difference between paired CT and VR measurements (0.58 [95% CI, to 2.74], ). Questionnaires showcased unanimous agreement on the user-friendly nature, effectiveness, and clinical value. The mixed reality application, validated through technical, preclinical, and clinical assessments, demonstrates precision and user-friendliness. Further research of our application is needed to validate the generalizability and impact on patient outcomes.

Unified Effort: Developing a Mixed Reality Serious Game for Collaborative Crisis Management Training

Abstract: Crises and disasters include both predictable and unpredictable events. In a turbulent world that has recently experienced a pandemic and wars, and is increasingly affected by climate change, there is a growing need to develop effective public crisis management capabilities. Emergency preparedness is a critical foundation for crisis management. Training and exercises are essential components of preparedness and are becoming increasingly relevant for a wide range of organisations in society. To achieve a unified response to threatening events, a wide range of actors - with different backgrounds, objectives, competencies and organisational frameworks - must overcome the challenges of working together. To address this complex situation, forums for knowledge sharing and learning are required. Serious games in mixed reality can be used as a platform for collaborative training and serve as such a forum. Mixed reality systems connect virtual and physical reality, providing users with a sense of immersion, engagement and interaction in both environments. The aim of this paper is to design a short, serious game for collaboration training with two command levels in a mixed reality. Theories of crisis management, along with related research on serious games design, create the framework for the game. The game content was further developed in a co-design process with selected professionals. This paper describes the development of the “Unified Effort” game. The serious game is designed for basic training of municipalities and county offices in soft skills such as communication, situational awareness and group decision making. The game is not intended to replace hard skills such as procedural training and field drills, but can be used as a supplement to this type of training.

Examining the Role of Augmented Reality and Virtual Reality in Safety Training

This study aims to provide a review of the existing literature regarding the use of extended reality technologies and the metaverse focusing on virtual reality (VR), augmented reality (AR), and mixed reality (MR) in safety training. Based on the outcomes, VR was predominantly used in the context of safety training with immersive VR yielding the best outcomes. In comparison, only recently has AR been introduced in safety training but with positive outcomes. Both AR and VR can be effectively adopted and integrated in safety training and render the learning experiences and environments more realistic, secure, intense, interactive, and personalized, which are crucial aspects to ensure high-quality safety training. Their ability to provide safe virtual learning environments in which individuals can practice and develop their skills and knowledge in real-life simulated working settings that do not involve any risks emerged as one of the main benefits. Their ability to support social and collaborative learning and offer experiential learning significantly contributed to the learning outcomes. Therefore, it was concluded that VR and AR emerged as effective tools that can support and enrich safety training and, in turn, increase occupational health and safety.

Understanding Mixed Reality Simulation as a Training Environment for Behavior and Special Education Knowledge and Skill Acquisition

Concerningly, pre-service teachers might utilize practices during practicum and student teaching experiences in ways that are less than satisfactory, or even harmful, to students, peers, or mentors. Mixed reality simulation (MRS), an educational tool for practicing teaching skills, may be a warranted approach to field experience that limits wrongdoing. However, few studies have researched the impact and meaningfulness of MRS teaching experiences. Therefore, this study combined both survey results and interview findings to report the extent to which MRS might improve the knowledge and skills of pre-service educators and whether MRS was considered an appropriate field experience alternative. Within one simulated scenario design, participants ( n = 94) were asked to engage five elementary-aged avatar students who presented challenging behaviors and asked questions or made comments about disability. Implications of findings are discussed in relation to mitigating common barriers of field experiences in higher education programs suggesting that, potentially, MRS may be an innovative way to respond to attrition concerns in the field of special education.

DIRA: A model of the user interface

The user interface is a central concept in human–computer interaction, but peculiarly fuzzy. This affects discussions of the fundamentals of our discipline and the positioning of our work. We propose a model of the user interface that consists of four elements: Devices, Interaction Techniques, Representations, and Assemblies (DIRA). We explain their roles in the user interface and discuss some associated concerns about evaluation and design. We then show how to use the model to describe the elements of user interfaces (with examples that include a menu, a fisheye interface, and notifications) and to analyze the central characteristics of user interface paradigms (including tangible user interfaces and mixed reality). Finally, we discuss how describing user interfaces with the model can drive their design and evaluation.

Using extended reality technologies in modern museums

This study explores the nature, core features, and evolution of extended reality (XR) technologies in the 21st century, focusing on their potential in the museum context. It examines academic research and representative examples of cultural heritage enhancement from international museum practices. The specific characteristics and properties of each technology within XR&amp;rsquo;s broader scope, including virtual reality (VR), augmented reality (AR), and mixed reality, are identified and analyzed. In addition, the study explores designs or case studies of these technologies&amp;rsquo; application in the museum environment, including the emerging Metaverse, which blends VR and AR within the same broader conceptual framework. Furthermore, the study examines why these innovative technologies resonate with museums and assesses their impact on relational communication between modern museums and the public, particularly younger audiences. It also highlights limitations or obstacles to their widespread adoption of XR technologies in museums, while outlining future prospects for their integration. Overall, the study provides a comprehensive overview of XR&amp;rsquo;s use in museums, focusing on how these technologies enhance cultural values. It offers reflection and suggestions for authors and museum professionals regarding the design and implementation of XR technologies in modern museums.