Collaborative Mixed Reality Research Articles

Supporting motion-capture acting with collaborative Mixed Reality

Technologies such as chroma-key, LED walls, motion capture (mocap), 3D visual storyboards, and simulcams are revolutionizing how films featuring visual effects are produced. Despite their popularity, these technologies have introduced new challenges for actors. An increased workload is faced when digital characters are animated via mocap, since actors are requested to use their imagination to envision what characters see and do on set. This work investigates how Mixed Reality (MR) technology can support actors during mocap sessions by presenting a collaborative MR system named CoMR-MoCap, which allows actors to rehearse scenes by overlaying digital contents onto the real set. Using a Video See-Through Head Mounted Display (VST-HMD), actors can see digital representations of performers in mocap suits and digital scene contents in real time. The system supports collaboration, enabling multiple actors to wear both mocap suits to animate digital characters and VST-HMDs to visualize the digital contents. A user study involving 24 participants compared CoMR-MoCap to the traditional method using physical props and visual cues. The results showed that CoMR-MoCap significantly improved actors’ ability to position themselves and direct their gaze, and it offered advantages in terms of usability, spatial and social presence, embodiment, and perceived effectiveness over the traditional method.

Pedagogy Designing With Augmented Reality: a Paradigm Shift in Educational Approaches

With Augmented Reality (AR) emerging as a revolutionary tool that reshapes conventional learning paradigms, the fusion of technology and education has given rise to creative educational approaches. This essay offers a thorough examination of augmented reality's place in education, highlighting its potential to improve pedagogy, engage students, and fundamentally alter the educational landscape. The essay starts by defining augmented reality as a link between the real and virtual worlds that provides an immersive and interactive learning experience. It explores the many uses of augmented reality, including anatomy classes with 3D interactive models and historical reenactments that take pupils back in time. These case studies show how augmented reality encourages participation, improves comprehension, and contextualizes learning materials. The fundamental ideas that underpin successful AR-enhanced schooling are presented in this study. It places a strong emphasis on scaffolding learning experiences, interaction, contextual relevance, and alignment with learning objectives. The difficulties of deploying AR are also discussed, giving educators ideas for a successful integration. These difficulties include technology infrastructure, educator training, content quality, and economic considerations. The research focuses on the potential of augmented reality to foster active learning. The relationship between augmented reality (AR) and active learning is examined, emphasizing the ways in which AR supports collaborative activities, problem-solving, and individualized learning paths. By imagining the role that augmented reality (AR) will play in future education, including personalized learning, virtual field excursions, mixed reality collaboration, and AI-driven adaptive learning, the essay demonstrates the revolutionary power of AR. The ethical challenges surrounding AR in the classroom are also covered, with a focus on the importance of addressing data protection, digital literacy, and responsible technology use. In order to prepare students for the challenges of a connected world while nurturing a lifetime love of learning, the study finishes by presenting augmented reality as a cornerstone of educational design. In essence, this article offers a thorough analysis of Augmented Reality's effects on education, highlighting its potential to transform pedagogy, improve learning opportunities, and influence the direction of education in the future. Educators are given a comprehensive grasp of how to use augmented reality to build immersive, successful learning environments by looking at case studies, design principles, difficulties, and future prospects

Mixed Reality collaboration environment improves the efficiency of human-centered industrial system: A case study in the mining industry

The emerging information and communication technologies (ICT) promote the intelligence of industrial systems. However, humans still play an important role in industrial systems, and they are required to collaborate in many scenarios to perform complex tasks. Mixed Reality (MR) technology allows the virtual-real fusion and real-time interaction, and has unparalleled advantages in building a collaborative environment. In this study, Theoretical and technical studies have been conducted to provide a universal solution for the industrial applications of MR collaboration. In terms of theory, a framework of the intelligent industrial system is first proposed by combining the human-centered idea and cloud-edge-terminal collaborative architecture. Based on this framework, the MR collaboration environment is then designed. Finally, the MR task scheduling model is developed to illustrate how the MR collaboration environment uses the cloud-edge-terminal collaborative architecture to improve the performance. The technical study is conducted based on the theoretical foundation. The key technologies involved in both co-located and remote collaboration modes, that can be achieved by the MR collaboration environment, are studied. In addition, the human–computer interaction (HCI) technology in MR collaboration is explored. Based on this study, the proposed MR collaboration environment is developed, tested, and evaluated by considering the mining industry as an example. Moreover, user surveys from a human-centered perspective are conducted. The results show that the MR collaboration environment has high applicability in multiple links in industrial scenarios represented by longwall mining. It can improve the quality, increase the efficiency, and reduce the costs. It can also provide a human-centered collaboration experience.

Study of Social Presence While Interacting in Metaverse with an Augmented Avatar during Autonomous Driving

In this paper, we studied the effects of using Microsoft HoloLens 2 in a Metaverse-based collaborative mixed reality environment on the driver’s social presence while using an autonomous driving system. In (semi-) autonomous vehicles the driver is the system’s monitor, and the driving process becomes a secondary task. Our approach is motivated by the advent of Microsoft Mesh XR technology that enables immersion in multi-person, shared mixed reality environments. We conducted a user study comparing the effects on social presence in two scenarios: baseline and mixed reality collaboration. During the baseline condition, participants communicated and interacted with another person using Skype/Meet which was installed on a mobile tablet. In the second scenario the participants used the Microsoft Mesh application installed on HoloLens 2 to collaborate in a mixed reality environment where each user is represented by an augmented 3D avatar. During the experiment, the participant had to perform a social interaction tell-a-lie task and a remote collaborative tic-tac-toe game, while also monitoring the vehicle’s behavior. The social presence was measured using the Harms and Biocca questionnaire, one of the most widely used tools for evaluating the user’s experience. We found that there are significant statistical differences for Co-presence, Perceived Emotional Interdependence, and Perceived Behavioral Interdependence, and participants were able to easily interact with the avatar in the mixed reality scenario. The proposed study procedure could be taken further to assess the driver’s performance during handover procedures, especially when the autonomous driving system encounters a critical situation.

The Impact of Sharing Gaze Behaviours in Collaborative Mixed Reality

In a remote collaboration involving a physical task, visualising gaze behaviours may compensate for other unavailable communication channels. In this paper, we report on a 360° panoramic Mixed Reality (MR) remote collaboration system that shares gaze behaviour visualisations between a local user in Augmented Reality and a remote collaborator in Virtual Reality. We conducted two user studies to evaluate the design of MR gaze interfaces and the effect of gaze behaviour (on/off) and gaze style (bi-/uni-directional). The results indicate that gaze visualisations amplify meaningful joint attention and improve co-presence compared to a no gaze condition. Gaze behaviour visualisations enable communication to be less verbally complex therefore lowering collaborators' cognitive load while improving mutual understanding. Users felt that bi-directional behaviour visualisation, showing both collaborator's gaze state, was the preferred condition since it enabled easy identification of shared interests and task progress.

Imitative Collaboration: A mirror-neuron inspired mixed reality collaboration method with remote hands and local replicas

Mixed reality can overlay and display 3D digital content in the real world, convey abstract concepts to users, and promote the understanding of complex tasks. However, the abstract graphics overlaid on the physical space may cause a certain cognitive load for local users and reduce the efficiency of collaboration. To improve the efficiency of remote collaboration, we conducted an elicitation study on assembly tasks, explored the user needs for collaboration, and defined the design goals of our remote collaboration method. Inspired by the mirror-neuron mechanism, we present an imitative collaboration method that allows local users to imitate the interaction behavior of remote users to complete tasks. We also propose a series of interaction methods for remote users to select, copy, and interact with the local point clouds to facilitate the expression of collaboration intentions. Finally, the results of a user study evaluating our imitative collaboration method on assembly tasks are reported, confirming that our method improves collaboration efficiency while reducing the cognitive load of local users.

A Study on Interaction Prediction for Reducing Interaction Latency in Remote Mixed Reality Collaboration

Various studies on latency in remote mixed reality collaborations (remote MR collaboration) have been conducted, but studies related to interaction latency are scarce. Interaction latency in a remote MR collaboration occurs because action detection (such as contact or collision) between a human and a virtual object is required for finding the interaction performed. Therefore, in this paper, we propose a method based on interaction prediction to reduce the time for detecting the action between humans and virtual objects. The proposed method predicts an interaction based on consecutive joint angles. To examine the effectiveness of the proposed method, an experiment was conducted and the results were given. From the experimental results, it was confirmed that the proposed method could reduce the interaction latency compared to the one obtained by conventional methods.

The effects of spatial auditory and visual cues on mixed reality remote collaboration

Collaborative Mixed Reality (MR) technologies enable remote people to work together by sharing communication cues intrinsic to face-to-face conversations, such as eye gaze and hand gestures. While the role of visual cues has been investigated in many collaborative MR systems, the use of spatial auditory cues remains underexplored. In this paper, we present an MR remote collaboration system that shares both spatial auditory and visual cues between collaborators to help them complete a search task. Through two user studies in a large office, we found that compared to non-spatialized audio, the spatialized remote expert’s voice and auditory beacons enabled local workers to find small occluded objects with significantly stronger spatial perception. We also found that while the spatial auditory cues could indicate the spatial layout and a general direction to search for the target object, visual head frustum and hand gestures intuitively demonstrated the remote expert’s movements and the position of the target. Integrating visual cues (especially the head frustum) with the spatial auditory cues significantly improved the local worker’s task performance, social presence, and spatial perception of the environment.

Augmented Virtual Teleportation for High-Fidelity Telecollaboration.

Telecollaboration involves the teleportation of a remote collaborator to another real-world environment where their partner is located. The fidelity of the environment plays an important role for allowing corresponding spatial references in remote collaboration. We present a novel asymmetric platform, Augmented Virtual Teleportation (AVT), which provides high-fidelity telepresence of a remote VR user (VR-Traveler) into a real-world collaboration space to interact with a local AR user (AR-Host). AVT uses a 360° video camera (360-camera) that captures and live-streams the omni-directional scenes over a network. The remote VR-Traveler watching the video in a VR headset experiences live presence and co-presence in the real-world collaboration space. The VR-Traveler's movements are captured and transmitted to a 3D avatar overlaid onto the 360-camera which can be seen in the AR-Host's display. The visual and audio cues for each collaborator are synchronized in the Mixed Reality Collaboration space (MRC-space), where they can interactively edit virtual objects and collaborate in the real environment using the real objects as a reference. High fidelity, real-time rendering of virtual objects and seamless blending into the real scene allows for unique mixed reality use-case scenarios. Our working prototype has been tested with a user study to evaluate spatial presence, co-presence, and user satisfaction during telecollaboration. Possible applications of AVT are identified and proposed to guide future usage.

Hand-in-Air (HiA) and Hand-on-Target (HoT) Style Gesture Cues for Mixed Reality Collaboration

In collaboration systems, collaborators often use hand gestures for showing information relating to a distant object. However, in viewer’s perspective, it is sometimes difficult to know where the conventional gesture cue (Hands-in-Air style: HiA) refers to because the conventional HiA gesture appears away from the distant object. In this paper, we investigate how two factors, distance to the object and view angle difference between collaborators, influence the understanding of HiA gesture by comparing the use of it at 25 positions with 5 distances and 5 view angles. In a user study, we found that the distance to the target object and view angle difference negatively influenced HiA gesture communication. The influence of the distance and view angle was more serious between smaller angles (0~30 degree) than larger angles (30~60 degree), and between shorter distances (1m $\sim 1.5\text{m}$ ) than between longer distance (1.5m $\sim 2\text{m}$ ). As a solution, we propose the Hands-on-Target (HoT) style which positions the hand gesture cue on the surface of the target object. The HoT style gesture cue dramatically reduces the negative effect of the distance and view angle difference. Participants completed the task 29.3 percent faster, selected the correct object 2.75 times more, and felt 63.1 percent less mental effort. For further investigation, we discuss on the extensibility of the HoT interface that it can be used not only for object selection but also for diverse type of gesture communication including gesture for object manipulation because the HoT interface can support all possible real world hand gestures.

Anatomy Studio: A tool for virtual dissection through augmented 3D reconstruction

3D reconstruction from anatomical slices allows anatomists to create three dimensional depictions of real structures by tracing organs from sequences of cryosections. However, conventional user interfaces rely on single-user experiences and mouse-based input to create content for education or training purposes. In this work, we present Anatomy Studio, a collaborative Mixed Reality tool for virtual dissection that combines tablets with styli and see-through head-mounted displays to assist anatomists by easing manual tracing and exploring cryosection images. We contribute novel interaction techniques intended to promote spatial understanding and expedite manual segmentation. By using mid-air interactions and interactive surfaces, anatomists can easily access any cryosection and edit contours, while following other user’s contributions. A user study including experienced anatomists and medical professionals, conducted in real working sessions, demonstrates that Anatomy Studio is appropriate and useful for 3D reconstruction. Results indicate that Anatomy Studio encourages closely-coupled collaborations and group discussion, to achieve deeper insights.

Intermediated Reality: A Framework for Communication Through Tele-Puppetry.

We introduce Intermediated Reality (IR), a framework for intermediated communication enabling collaboration through remote possession of entities (e.g., toys) that come to life in mobile Mediated Reality (MR). As part of a two-way conversation, each person communicates through a toy figurine that is remotely located in front of the other participant. Each person's face is tracked through the front camera of their mobile devices and the tracking pose information is transmitted to the remote participant's device along with the synchronized captured voice audio, allowing a turn-based interactive avatar chat session, which we have called ToyMeet. By altering the camera video feed with a reconstructed appearance of the object in a deformed pose, we perform the illusion of movement in real-world objects to realize collaborative tele-present augmented reality (AR). In this turn based interaction, each participant first sees their own captured puppetry message locally with their device's front facing camera. Next, they receive a view of their counterpart's captured response locally (in AR) with seamless visual deformation of their local 3D toy seen through their device's rear facing camera. We detail optimization of the animation transmission and switching between devices with minimized latency for coherent smooth chat interaction. An evaluation of rendering performance and system latency is included. As an additional demonstration of our framework, we generate facial animation frames for 3D printed stop motion in collaborative mixed reality. This allows a reduction in printing costs since the in-between frames of key poses can be generated digitally with shared remote review.

Revisiting collaboration through mixed reality: The evolution of groupware

Abstract Collaborative Mixed Reality (MR) systems are at a critical point in time as they are soon to become more commonplace. However, MR technology has only recently matured to the point where researchers can focus deeply on the nuances of supporting collaboration, rather than needing to focus on creating the enabling technology. In parallel, but largely independently, the field of Computer Supported Cooperative Work (CSCW) has focused on the fundamental concerns that underlie human communication and collaboration over the past 30-plus years. Since MR research is now on the brink of moving into the real world, we reflect on three decades of collaborative MR research and try to reconcile it with existing theory from CSCW, to help position MR researchers to pursue fruitful directions for their work. To do this, we review the history of collaborative MR systems, investigating how the common taxonomies and frameworks in CSCW and MR research can be applied to existing work on collaborative MR systems, exploring where they have fallen behind, and look for new ways to describe current trends. Through identifying emergent trends, we suggest future directions for MR, and also find where CSCW researchers can explore new theory that more fully represents the future of working, playing and being with others.

Cyber Security Threats and Challenges in Collaborative Mixed-Reality

Collaborative Mixed-Reality (CMR) applications are gaining interest in a wide range of areas including games, social interaction, design and health-care. To date, the vast majority of published work has focused on display technology advancements, software, collaboration architectures and applications. However, the potential security concerns that affect collaborative platforms have received limited research attention. In this position paper, we investigate the challenges posed by cyber-security threats to CMR systems. We focus on how typical network architectures facilitating CMR and how their vulnerabilities can be exploited by attackers, and discuss the degree of potential social, monetary impacts, psychological and other harms that may result from such exploits. The main purpose of this paper is to provoke a discussion on CMR security concerns. We highlight insights from a cyber-security threat modelling perspective and also propose potential directions for research and development toward better mitigation strategies. We present a simple, systematic approach to understanding a CMR attack surface through an abstraction-based reasoning framework to identify potential attack vectors. Using this framework, security analysts, engineers, designers and users alike (stakeholders) can identify potential Indicators of Exposures (IoE) and Indicators of Compromise (IoC). Our framework allows stakeholders to reduce their CMR attack surface as well understand how Intrusion Detection System (IDS) approaches can be adopted for CMR systems. To demonstrate the validity to our framework, we illustrate several CMR attack surfaces through a set of use-cases. Finally, we also present a discussion on future directions this line of research should take.

A Comparison Between Two Different Approaches for a Collaborative Mixed-Virtual Environment in Industrial Maintenance.

Nowadays the market is becoming increasingly competitive, factories are required not only to enhance the product quality but also to reduce manufacturing and maintenance times. In an industrial context, modern factories are composed by many automated systems, such as industrial robots, which can perform different tasks. Although industrial robots are becoming more powerful and efficient, human workers are still required to accomplish different operations, such as training and maintenance procedures. The proposed research aims to assess a remote interaction system in an industrial training collaborative mixed-reality (CMR) environment. A remote expert user is capable of explaining a training procedure to an unskilled local user. Remote and local users interact using different interaction systems: the remote operator gives assistance using an immersive Virtual Reality (VR) device, whereas the local user interacts using a wearable Augmented Reality (AR) device. A comparison between an interaction based on the presence of a virtual human and one based on the use of abstract icons is proposed. In the first case, a virtual 3D representation of the remote technician is shown to the local user by using AR: the remote technician can pinpoint the components involved in the training procedure and the local user can visualize the instructions through some animations of the virtual avatar. In the second case, the local user cannot see a 3D representation of the remote technician; on the other hand, different 3D models, such as animated icons, are displayed to the local operator through AR depending on the component pinpointed by the remote technician in the virtual environment. Each 3D icon should suggest to the local user which component has to be manipulated at the current step of the procedure. Preliminary results suggest that the interface that requires less resources to be developed and managed should be preferred. Although in no audio condition the virtual avatar may improve the sense of presence of the remote technician, the use of abstract metaphors seems to be of primary importance to successfully complete an industrial task.

The Effects of Sharing Awareness Cues in Collaborative Mixed Reality.

Augmented and Virtual Reality provide unique capabilities for Mixed Reality collaboration. This paper explores how different combinations of virtual awareness cues can provide users with valuable information about their collaborator's attention and actions. In a user study (n = 32, 16 pairs), we compared different combinations of three cues: Field-of-View (FoV) frustum, Eye-gaze ray, and Head-gaze ray against a baseline condition showing only virtual representations of each collaborator's head and hands. Through a collaborative object finding and placing task, the results showed that awareness cues significantly improved user performance, usability, and subjective preferences, with the combination of the FoV frustum and the Head-gaze ray being best. This work establishes the feasibility of room-scale MR collaboration and the utility of providing virtual awareness cues.

A systematic review of the current state of collaborative mixed reality technologies: 2013–2018

Over the last few decades, Mixed Reality (MR) interfaces have received great attention from academia and industry. Although a considerable amount of research has already been done to support collaboration between users in MR, there is still no systematic review to determine the current state of collaborative MR applications. In this paper, collaborative MR studies published from 2013 to 2018 were reviewed. A total of 259 papers have been categorised based on their application areas, types of display devices used, collaboration setups, and user interaction and experience aspects. The primary contribution of this paper is to present a high-level overview of collaborative MR influence across several research disciplines. The achievements from each application area are summarised. In addition, remarkable papers in their respective areas are highlighted. Among other things, our study finds that there are three complementary factors to support and enhance collaboration in MR environments: (i) annotation techniques, which provide non-verbal communication cues to users, (ii) cooperative object manipulation techniques, which divide complex 3D object manipulation process into simpler tasks between different users, and (iii) user perception and cognition studies, which aim to lessen cognitive workload for task understanding and completion, and to increase users’ perceptual awareness and presence. Finally, this paper identifies research gaps and future directions that can be useful for researchers who want to explore ways on how to foster collaboration between users and to develop collaborative applications in MR.

Empowerment and embodiment for collaborative mixed reality systems

AbstractWe present several mixed‐reality‐based remote collaboration settings by using consumer head‐mounted displays. We investigated how two people are able to work together in these settings. We found that the person in the AR system will be regarded as the “leader” (i.e., they provide a greater contribution to the collaboration), whereas no similar “leader” emerges in augmented reality (AR)‐to‐AR and AR‐to‐VRBody settings. We also found that these special patterns of leadership only emerged for 3D interactions and not for 2D interactions. Results about the participants' experience of leadership, collaboration, embodiment, presence, and copresence shed further light on these findings.

Immersive Mixed Reality for Manufacturing Training

In the complex manufacturing sector a considerable amount of resources is focused on training workers and developing new skills. Increasing the effectiveness of those processes and reducing the investment required is an outstanding issue. In this paper, we present an experiment (n=20) that shows how modern metaphors such as collaborative Mixed Reality can be used to transmit procedural knowledge and could eventually replace other forms of face-to-face training. We implemented a Mixed Reality setup with see-through cameras attached to a Head Mounted Display. The setup allowed for real-time collaborative interactions and could simulate conventional forms of training. We tested the system implementing a manufacturing procedure of an aircraft maintenance door. The obtained results indicate that performance levels in the Immersive Mixed Reality training were not significantly different than in the conventional face-to-face training condition. These results and their implications for future training and the use of Virtual Reality, Mixed Reality and Augmented Reality paradigms in this context are discussed in this paper.

協調型複合現実空間における視線認知特性を考慮した視線提示(「複合現実感3」特集)

協調型複合現実空間における視線認知特性を考慮した視線提示(「複合現実感3」特集)