Future Mixed Reality Research Articles

Advancing Mixed Reality Game Development: An Evaluation of a Visual Game Analytics Tool in Action-Adventure and FPS Genres

In response to the unique challenges of Mixed Reality (MR) game development, we developed GAMR , an analytics tool specifically designed for MR games. GAMR aims to assist developers in identifying and resolving gameplay issues effectively. It features reconstructed gameplay sessions, heatmaps for data visualization, a comprehensive annotation system, and advanced tracking for hands, camera, input, and audio, providing in-depth insights for nuanced game analysis. To evaluate GAMR 's effectiveness, we conducted an experimental study with game development students across two game genres: action-adventure and first-person shooter (FPS). The participants used GAMR and provided feedback on its utility. The results showed a significant positive impact of GAMR in both genres, particularly in action-adventure games. This study demonstrates GAMR 's effectiveness in MR game development and suggests its potential to influence future MR game analytics, addressing the specific needs of developers in this evolving area.

Visualizing Hand Force With Wearable Muscle Sensing for Enhanced Mixed Reality Remote Collaboration.

In this paper, we present a prototype system for sharing a user's hand force in mixed reality (MR) remote collaboration on physical tasks, where hand force is estimated using wearable surface electromyography (sEMG) sensor. In a remote collaboration between a worker and an expert, hand activity plays a crucial role. However, the force exerted by the worker's hand has not been extensively investigated. Our sEMG-based system reliably captures the worker's hand force during physical tasks and conveys this information to the expert through hand force visualization, overlaid on the worker's view or on the worker's avatar. A user study was conducted to evaluate the impact of visualizing a worker's hand force on collaboration, employing three distinct visualization methods across two view modes. Our findings demonstrate that sensing and sharing hand force in MR remote collaboration improves the expert's awareness of the worker's task, significantly enhances the expert's perception of the collaborator's hand force and the weight of the interacting object, and promotes a heightened sense of social presence for the expert. Based on the findings, we provide design implications for future mixed reality remote collaboration systems that incorporate hand force sensing and visualization.

Mixed Reality in Multiuser Participatory Design: Case Study of the Design of the 2022 Nordic Pavilion Exhibition at the Venice Biennale

The case study documents the design process of the physical and digital versions of the heritage-valued Nordic pavilion at the Venice Biennale. The case facilitated a multiuser collaboration in mixed reality (MR), studying the technologies’ influence upon user interactions and design decision making. Retitled as the ‘Sami Pavilion’ in tribute to the Sami artists from Norway, Sweden, and Finland, which the exhibition featured, the case study took place between 2019 and 2022, primarily during the COVID-19 pandemic. The context of the case study prompted a need to explore MR methods to overcome travel restrictions. While MR has shown some interesting utility in design research, the literature indicates the need for more concrete case work. It also was necessary to design a custom solution for multiuser collaboration. As the transferability of predictive design decisions in MR to the physical building relied upon replication between user experiences, the case embodied interesting challenges to prevalent Nordic architectural theory, particularly that of ‘genius loci’ or the ‘spirit of place’, which was a strong component of the heritage value of the building and, thus, an important design narrative. The case study documents how artworks and positions of artworks were tested in various configurations within the MR model by curators and designers to simulate the spatial experience of the design options. Several key design decisions were made based on the unique vantage points offered in MR. The MR model was then used to generate 2D technical documentation and installation instructions, which were installed on site. Studies to check the relationship between the MR model and the finished, physical result were conducted. Findings depicted a high degree of transferability between the MR model and the physical exhibition while noting discrepancies between the field of view (FOV) in the MR and physical spaces in which objects felt smaller in the real-life, built result. Possibilities and limitations for future MR implementation in the design and construction of complex projects in heritage situations are discussed in addition to implications for current architectural theory regarding place and experience from a non-dualistic perspective.

Single-camera 3D head fitting for mixed reality clinical applications

We address the problem of estimating the shape of a person’s head, defined as the geometry of the complete head surface, from a video taken with a single moving camera, and determining the alignment of the fitted 3D head for all video frames, irrespective of the person’s pose. 3D head reconstructions commonly tend to focus on perfecting the face reconstruction, leaving the scalp to a statistical approximation. Our goal is to reconstruct the head model of each person to enable future mixed reality applications. To do this, we recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape by iteratively fitting a 3D morphable model of the head with the dense reconstruction in canonical space and fitting it to each person’s head, using both traditional facial landmarks and scalp features extracted from the head’s segmentation mask. Our approach recovers consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of environments from living rooms to outdoor spaces.

Virtual Reality in Healthcare Skills Training: The Effects of Presence on Acceptance and Increase of Knowledge

Abstract With an ever-increasing need of skilled healthcare workers, efficient learning methods like Virtual Reality (VR) are becoming increasingly important. We developed and tested a VR simulation for endotracheal suctioning. The aim of this pilot study was to examine the VR simulation’s acceptance and increase of knowledge among participants. Furthermore, the effects of presence on acceptance and increase of knowledge were investigated. A total of 51 students participated in the pilot study, using a quasi-experimental pre-post-test design. A modified Unified Theory of Acceptance and Use of Technology (UTAUT) and the Igroup Presence Questionnaire (IPQ) were used. Correlation and regression analyses were performed. Pre- and post-tests showed a significant increase of knowledge (p < 0.001). The correlation between presence and behavioural intention was highly positive (r = 0.52, p < 0.001). Performance and effort expectancy are dominant effects on behavioural intention of using the VR simulation as an educational tool. The results indicate that a simulation which conveys a higher sense of presence is more likely to be accepted by learners. Regarding outcomes of presence on increase of knowledge, we found no significant correlation. Based on our study, we propose a design for a future mixed reality simulation with haptic elements and a plan on how to assess skills improvement.

Eyes-free Target Acquisition During Walking in Immersive Mixed Reality.

Reaching towards out-of-sight objects during walking is a common task in daily life, however the same task can be challenging when wearing immersive Head-Mounted Displays (HMD). In this paper, we investigate the effects of spatial reference frame, walking path curvature, and target placement relative to the body on user performance of manually acquiring out-of-sight targets located around their bodies, as they walk in a spatial-mapping Mixed Reality (MR) environment wearing an immersive HMD. We found that walking and increased path curvature negatively affected the overall spatial accuracy of the performance, and that the performance benefited more from using the torso as the reference frame than the head. We also found that targets placed at maximum reaching distance yielded less error in angular rotation and depth of the reaching arm. We discuss our findings with regard to human walking kinesthetics and the sensory integration in the peripersonal space during locomotion in immersive MR. We provide design guidelines for future immersive MR experience featuring spatial mapping and full-body motion tracking to provide better embodied experience.

Research on Battlefield Situation Display Based on Mixed Reality Technology

Aiming at the problem that the current combat command and control system adopts the two-dimensional information display method, which is not intuitive and easy to overlap, and limits the decision potential of the operator, a method of applying hybrid display technology to battlefield command decision is proposed. By analyzing the application characteristics of mixed reality technology in situation display and command control, using 3D Max, Unity to create virtual models and space environment, writing control scripts such as gaze control and gesture operation under VS2017 platform, and implementing applications based on Hololens mixed reality helmet. The battlefield situation display effect of projecting the holographic image onto the real world surface and the command and control mode of the operator interacting with the holographic image in various forms are realized, which proves the possibility that the future mixed reality technology is applied to actual combat command.

An Investigation of Collective Human Behavior in Large-Scale Mixed Reality Spaces

Future mixed reality systems will need to support large numbers of simultaneous, nonexpert users at reasonable per-user costs if the systems are to be widely deployed within society in the short to medium term. We have constructed a prototype of such a system, an interactive entertainment space called Ada that was designed to behave like a simple organism. Using Ada we conducted two studies: the first assessing the effect of varying the operating parameters of the space on the collective behavior and attitudes of its users, and the second assessing the relationships among user demographics, behavior, and attitudes. Our results showed that small changes in the ambient settings of the environment have a significant effect on both user attitudes and behavior, and that the changes in user attitudes do not necessarily correspond to the environmental changes. We also found that individual user opinions are affected by demographics and reflected in overt behavior. Using these results, we propose some tentative guidelines for the design of future shared mixed reality spaces.