Mixed Reality Game Research Articles

Advancing Mixed Reality Game Development: An Evaluation of a Visual Game Analytics Tool in Action-Adventure and FPS Genres

In response to the unique challenges of Mixed Reality (MR) game development, we developed GAMR , an analytics tool specifically designed for MR games. GAMR aims to assist developers in identifying and resolving gameplay issues effectively. It features reconstructed gameplay sessions, heatmaps for data visualization, a comprehensive annotation system, and advanced tracking for hands, camera, input, and audio, providing in-depth insights for nuanced game analysis. To evaluate GAMR 's effectiveness, we conducted an experimental study with game development students across two game genres: action-adventure and first-person shooter (FPS). The participants used GAMR and provided feedback on its utility. The results showed a significant positive impact of GAMR in both genres, particularly in action-adventure games. This study demonstrates GAMR 's effectiveness in MR game development and suggests its potential to influence future MR game analytics, addressing the specific needs of developers in this evolving area.

Toward next generation mixed reality games: a research through design approach

Mixed reality (MR) games refer to games that integrate physical entities with digitally mediated contents. Currently, it entails game creators to integrate heterogeneous virtual and physical components, which is often time-consuming and labor-intensive, without the support of a coherent technology stack. The underlying technodiversity manifested by the research corpus suggests a complicated, multi-dimensional design space that goes beyond merely technical concerns. In this research, we adopted a research-through-design approach and proposed an MR game technology stack that facilitates flexible, low-code game development. As design grounding, we first surveyed 34 state-of-the-art studies, and results were synergized into three different spectra of technological affordances, respectively activity range, user interface and feedback control, to inform our next design process. We then went through an iterative prototyping phase and implemented an MR game development toolset. A co-design workshop was conducted, where we invited 15 participants to try the prototype tools and co-ideate the potential use scenarios for the proposed technology stack. First-hand user feedback was collected via questionnaires and semi-structured interviews. As a result, four conceptual game designs with three major design implications were generated, which conjointly reflect a broader understanding on MR gameful experience and contribute fresh insights to this emerging research domain.

The Ethics of Mixed Reality Games

The Ethics of Mixed Reality Games

Optimizing Camera Setup for In-Home First-Person Rendering Mixed-Reality Gaming System

With the advent of 3D humanoid reconstruction techniques, using a realistic 3D human avatar in a serious game has become popular. This realistic representation in the virtual environment could be achieved using a single low-cost RGB-D camera such as Kinect. However, properly setting up such a camera system for high-quality rendering can be challenging due to the relatively restricted in-home environment. In this paper, we address the challenge of finding optimized camera setup guidelines for an in-home first-person perspective mixed reality (MR) gaming system. We use an MR system with personalized humanoids to simulate the texture reconstruction for a user under a specific camera configuration. Then, a derivative-free optimization is leveraged as a black box approach to search for the optimized camera setup through iterative simulation. We also introduce a novel skeleton-based calibration to address the effects of physically varying the camera's position. For evaluation, two experiments are carried out to evaluate the correctness and effectiveness of the proposed calibration. Furthermore, we conduct a case study using simulation-based optimization for reconstructing lower limb amputees. This work can potentially help locate a proper camera setup for an MR system within the constraints of an in-home environment.

Teaming with a Robot in Mixed Reality: Dynamics of Trust, Self-Efficacy, and Mental Models Affected by Information Richness

This study employs an innovative mixed-reality game to investigate trust, self-efficacy, and collaboration willingness with a mobile robot in relation to information and mental models. Previous research has demonstrated the significant impact of introductory information on technology use and trust, which further evolves over the course of interactions. Expanding on this, we explore new facets of information content through comprehensive measurements in two empirical studies: In study 1, 68 participants watched one of two video tutorials (high or low information richness) before collaborating with a physical robot in a shared virtual 3D environment. Repeated measures served to investigate temporal dynamics. In study 2, 37 participants additionally engaged in qualitative interviews to better understand how their mental models changed with increasing robot experience. Both tutorials were found to influence user trust and self-efficacy over time, with no significant differences between the two variants. Users valued information on how to communicate with the robot, but still relied on trial-and-error approaches. We argue for a moderating effect of information content and richness and highlight the need for carefully designed tutorials to enhance human-robot interaction.

Museum Games and Personal Epistemology: A Study on Students’ Critical Thinking with a Mixed Reality Game

This study examines how 12-15-year-old students use information while playing Geome, a mixed reality game intended for museum school visits. Geome presents environmental issues, prompting students and asking them to discern and dismiss misinformation and rumors. The study aims to analyze the students' playful learning experience and their perspective on the game. It focuses on the students' critical thinking, interactions and beliefs about knowledge and knowing, referred to as personal epistemology. Adopting a qualitative approach, the research was conducted across three classes in a museum. A combination of audio, video, and in-game interactions was collected from specific moments during gameplay and analyzed according to epistemological dimensions (Certainty, Simplicity, Source, Justification). Video analysis suggests that when faced with ill-structured problems within a playful scenario, some students are spurred to actively process information and develop critical thinking skills. Meanwhile others remain entrenched in their initial conceptions about the nature of knowledge and the act of knowing. The study discusses how the game's characteristics shape students' personal epistemology. Overall, this research demonstrates that games in museum contexts have the potential to promote active learning and critical thinking in some students, when confronted with complex or ill-structured problems.

Creating Location-Based Augmented Reality Games and Immersive Experiences for Touristic Destination Marketing and Education

The aim of this paper is to present an approach that utilizes several mixed reality technologies for touristic promotion and education. More specifically, mixed reality applications and games were created to promote the mountainous areas of Western Macedonia, Greece, and to educate visitors on various aspects of these destinations, such as their history and cultural heritage. Location-based augmented reality (AR) games were designed to guide the users to visit and explore the destinations, get informed, gather points and prizes by accomplishing specific tasks, and meet virtual characters that tell stories. Furthermore, an immersive lab was established to inform visitors about the region of interest through mixed reality content designed for entertainment and education. The lab visitors can experience content and games through virtual reality (VR) and augmented reality (AR) wearable devices. Likewise, 3D content can be viewed through special stereoscopic monitors. An evaluation of the lab experience was performed with a sample of 82 visitors who positively evaluated features of the immersive experience such as the level of satisfaction, immersion, educational usefulness, the intention to visit the mountainous destinations of Western Macedonia, intention to revisit the lab, and intention to recommend the experience to others.

Digital storytelling and hopeful last chance tourism experiences

Last chance tourism, which can be described as the practice of tourism in environmentally damaged or threatened spaces, is an emerging trend that has been considered problematic due to the pressure that it places on already fragile environments. This article explores the possibilities of turning last chance tourism experiences into hopeful tourism experiences, by creating emotionally driven last chance tourism experiences in controlled and safe environments through digital storytelling. We draw on literature on transformative, hopeful and flourishing tourism to argue that last chance experiences provide unique opportunities for eliciting reflection among tourists and promote a greater environmental awareness if they are driven by emotionally engaging narratives. Our study explores the outcomes of an experiment conducted at the Ajuda Botanical Garden (Lisbon, Portugal) with a futuristic mixed-reality game that places users as members of a scientific expedition looking for plants that have become extinct in the twenty-second century. The results of this study show how the game’s last chance narrative generated contradictory, mixed feelings among the participants, which ultimately led to hopeful existential reflection about nature conservation action. With this in mind, we reflect on the potential of transformative last chance tourism experiences guided by digital narratives.

Research on User Experience Model Based on Mixed Reality Interactive Games

This paper proposes a user experience model based on mixed reality interactive games. The model combines the characteristics of user experience and mixed reality games, and through user surveys and the establishment of a user experience model, it verifies that sensory experience, interactive experience, emotional experience, and behavioral experience are the main factors in mixed reality games, which can provide guidance for the design of mixed reality games.

A Serious Mixed-Reality Game for Training Police Officers in Tagging Crime Scenes

Recognizing and collecting evidence at a crime scene are essential tasks for gathering information about perpetrators and/or the dynamics of a criminal event. Hence, the success of a crime investigation is strongly based on the ability of forensic investigators to perform these tasks. Recent studies observing and comparing the performance of experts and novices have highlighted the importance of experience and training for search and recovery strategies at crime scenes. Therefore, relevant training programs in evidence-recovery techniques should be attended by novices to improve their skills. However, the knowledge transfer between skills acquired in the classroom and their practical application in the field is a challenging task. In order to relieve this problem, this paper proposes a serious mixed-reality game, which is called TraceGame, aiming to support the training activities of novice forensic investigators by improving their skills related to the search and recovery of evidence at crime scenes. The purpose of the game is to identify the greatest number of useful traces present in a crime scene that is physically reconstructed at the training site as quickly as possible. As shown in an experimental session, TraceGame is a promising tool for supporting the training of novice forensic investigators.

Innovative Mixed Reality Based Rehabilitative Tool for Upper Limb Injuries Utilizing HoloLens 2

<h3>Research Objectives</h3> To develop a more interactive method of at-home upper limb rehabilitation - utilizing the HoloLens 2 - that allows therapists to provide rehab exercises and track their patients' progress by recording the patients' upper limb joint angles. <h3>Design</h3> Experimental study. <h3>Setting</h3> This experiment was set in a laboratory (BioRobotics Lab) environment. <h3>Participants</h3> Ten healthy adult participants (Mean age: 28.7 years) with no prior major arm-related injuries. <h3>Interventions</h3> Upper arm rehabilitation exercise through gamification in a mixed reality environment. <h3>Main Outcome Measures</h3> We assessed the functionality and the usability of the rehabilitative tool in performing different rehab exercises for upper limb injuries. <h3>Results</h3> The system successfully provided rehab exercises for upper limb injuries through interactive mixed reality games. The system can mimic the upper limb behavior without additional sensors during the rehab sessions. We also observed that the different upper limb joints' degrees of freedom are independent of each other. <h3>Conclusions</h3> Results indicated that the developed system could provide rehab exercises to individuals with upper limb injuries in a more interactive environment than the existing systems. Our next step will be to test the system in a clinical setting. <h3>Author(s) Disclosures</h3> The authors have no conflicts of interest to declare.

Gamification of Upper Limb Rehabilitation in Mixed-Reality Environment

The advancements in mixed reality (MR) technology in recent years have provided us with excellent prospects for creating novel approaches to supplement conventional physiotherapy to maintain a sufficient quantity and quality of rehabilitation. The use of MR systems to facilitate patients’ participation in intensive, repetitive, and task-oriented practice using cutting-edge technologies to enhance functionality and facilitate recovery is very encouraging. Multiple studies have found that patients who undergo therapy using MR experience significant improvements in upper limb function; however, assessing the efficacy of MR is challenging due to the wide variety of methods and tools used. Because of these challenges, a novel approach, gamified MR-based solution for upper extremity rehabilitation, is proposed, which is an MR application for the Microsoft HoloLens 2, complete with game levels, and can measure the ranges of motion of the arm joints. The proposed rehabilitative system’s functionality and usability were evaluated with ten healthy adult participants with no prior arm-related injuries and two occupational therapists (OTs). The system successfully provided rehab exercises for upper limb injuries through interactive mixed-reality games. The system can mimic upper limb behavior without additional sensors during rehab sessions. Unlike previously researched technologically-based rehabilitation methods, this method can integrate arm–joint data within the application and are independent of one another. The results and comparisons show that this system is relevant, accurate, and superior to previous VR-based rehabilitation methods because the VR-based system is blind to the surroundings, whereas the proposed approach has spatial awareness of the environment.

Toward a Digital Companion to Monitor a Mixed Reality Game

The game orchestration by the game master during a game-based learning session requires to (1) collect data from player interactions, (2) visualize indicators built with this data, (3) take decisions regarding the monitoring of the activity and (4) interact with players. The complexity of this monitoring increases with mixed-reality games due to the multiplicity of possible interactions with tangible and digital game elements. This article deals with playing analytics and the design of a digital companion to help a game master to orchestrate Geome, a mixed-reality game dedicated to museum school visits. The prototype is the result of a co-design process between researchers, practitioners, and computer scientists. This work allowed us to (1) define the interactions established during the use of the game that should be traced for the game master and (2) test the prototype with 3 classes (12-15 years old). We found that, while the players play the game as intended using most of the features, the game master makes little use of his dashboard and doesn’t interact with players through the dedicated interface.

Back to the Control Room: Managing Artistic Work

Control rooms have long been a key domain of investigation in HCI and CSCW as sites for understanding distributed work and fragmented settings, as well as the role and design of digital technologies in that work. Although research has tended to focus mainly on ‘command and control’ configurations, such as rail transport, ambulance dispatch, air traffic and CCTV rooms, centres of coordination shaped by artistic and performative concerns have much to contribute. Our study examines how a professional team of artists and volunteers stage manage and direct the performance of a mixed reality game from a central control room, with remote runners performing live video streaming from the streets nearby to online players. We focus on the work undertaken by team members to bring this about, exploring three key elements that enable it. First, we detail how team members oriented to the work as an artistic performance produced for an audience, how they produced compelling, varied content for online players, and how the quality of the work was ongoingly assessed. Second, we unpack the organisational hierarchy in the control room’s division of labour, and how this was designed to manage the challenges of restricted informational visibility there. Third, we explore the interactional accomplishment of the performance by looking at the role of radio announcements from the event’s director to orchestrate how the performance developed over time. Announcements were used to resolve trouble and provide instructions for avoiding future performative problems; but more centrally, to give artistic direction to runners in order to shape the performance itself. To close we discuss how this study of a performance impacts CSCW’s understandings of control room work, how the problem of ‘diffuse’ tasks like artistic work is co-ordinated, and how orientations towards quality as an artistic concern is manifest in / as control room practices. We also reflect on hierarchical and horizontal control room arrangements, and the role of video as both collaborative resource and product.Graphical abstract

Notes from Archive to AR: towards a creative knowledge exchange

Archive to AR was a knowledge exchange project that took place during 2019 and 2020, and which was funded in its second phase by an XR Stories Small Collaborative R&amp;D grant. A collaboration between the University of Sheffield, Sheffield-based digital agency Hive IT, and the National Railway Museum, its aim was to develop an idea and partial prototype for a mixed-reality game based on the National Railway Museum’s archive, which could be played on a phone or a tablet. Drawing on my experience as a practitioner-academic working on Archive to AR, the aim of this article is to present and reflect upon the creative practice-related methodological outcomes of the project. My intention is to consider which principles for doing knowledge exchange (KE) emerge when embodied, creative process is prioritised and, in line with a practice-as-research approach, the KE space is cast as a site of knowledge production as well as exchange. Part provocation, part ethno-autographic reflection and part analysis, this paper’s overall aim is to contribute to the ongoing conversation about how KE and creative work can overlap within a new ‘third space’ for academia in a way that prioritises process and method, but which is nevertheless conducive to high-quality artistic outputs.

Maturing the narrative sense through mixed reality games

This article explores the human ability to see meaningful narratives in the world and to invent fictional narratives as a sense. The senses allow us to pick up on patterns in the world and to coordinate our behaviour in relationship with those patterns. Sensing narratives plays an important role in situational appraisals and provides important context for what we see, hear, touch, etc. Drawing upon the predictive processing framework in cognitive science, this article will defend this thesis and explore how virtual and mixed reality technologies can contribute to new ways of making sense of the world. These new tools of situational appraisal will be understood in the context of a cultural and cognitive condition that media theories Douglas Rushkoff calls ‘present shock’.

Conventional Cervical Exercises Compared with a Mixed-Reality-Based Game in Asymptomatic Subjects: An Exploratory Crossover Pilot Study

Mixed reality presents itself as a potential technological tool for the management of people with musculoskeletal disorders, without having as many adverse side effects as immersive virtual reality. The objective of this study was to explore the possibilities of a mixed-reality game, performing task-oriented cervical exercises compared to conventional therapeutic exercises in sensorimotor outcome measures in asymptomatic subjects. A randomized crossover pilot study was performed with two intervention groups: a mixed-reality group (MRG) and a conventional exercise group (CEG). The cervical joint position error test (CJPET) and deep cervical flexor endurance test (DCFET) were measured as sensorimotor outcomes. Statistically significant differences were found in the pre–post comparison in the DCFET for both groups (MRG: t = −3.87, p &lt; 0.01; CEG: t = −4.01, p &lt; 0.01) and in the extension of the CJPET for the MRG (t = 3.50, p &lt; 0.01). The rest of the measurements showed no significant differences comparing both groups pre- and postintervention (p &gt; 0.05). Mixed reality has apparently the same positive effects as conventional exercises in sensorimotor outcomes in asymptomatic subjects. These results could help in future studies with mixed virtual reality in the management of people with musculoskeletal disorders.

A Systematic Literature Review of Virtual, Augmented, and Mixed Reality Game Applications in Healthcare

Virtual reality, augmented reality, and mixed reality (VR/AR/MR) as information and communication technologies have been recognised and implemented in healthcare in recent years. One of the popular application ways is games, due to the potential benefits of providing an engaging and immersive experience in a virtual environment. This study presents a systematic literature review that evaluates the state-of-the-art on VR/AR/MR game applications in healthcare by collecting and analysing related journal and conference papers published from 2014 through to the first half of 2020. After retrieving more than 3,000 papers from six databases, 88 articles, from both computer science and medicine, were selected and analysed in the review. The articles are classified and summarised based on their (1) publication information, (2) design, implementation, and evaluation, and (3) application. The presented review is beneficial for both researchers and developers interested in exploring current research and future trends in VR/AR/MR in healthcare.

Geo-Based Mixed Reality Gaming Market Analysis

Mixed reality has constantly been a symbol of the future for many years because it integrates many related areas of art and science to produce superior interactive experiences. Recently, it has become tremendously popular with location-based features that allow a new immersive interaction between people. In this paper, we present a novel research of pioneering geo-based mixed reality games, including our own research, to explore the evolution of mixed reality games using location-based features. Five major geo-based mixed reality games are selected for analysis based on the periods built: Human Pacman, AR Car Game, Ingress, Pokémon Go, and AR Mario Kart Live. The geo-based mixed reality gaming market analysis is explored extensively using the semiqualitative method to discover the relationships between the independent variable (year built) and the dependent variables (number of academic paper downloads, Google Trends, revenues, and number of downloads). We then compare each game based on the year built, the devices used in the game, the game purposes, the number of academic paper downloads, the revenues, the number of downloads, and the types of virtual items in the game, to deeply understand the evolution of geo-based mixed reality games. Finally, recommendations for future directions are included.

SigSense: Mobile Crowdsensing Based Incentive Aware Geospatial Signal Monitoring for Base Station Installation Recommendation Using Mixed Reality Game

SigSense: Mobile Crowdsensing Based Incentive Aware Geospatial Signal Monitoring for Base Station Installation Recommendation Using Mixed Reality Game