Mixed Reality Remote Collaboration Research Articles

A novel mixed reality remote collaboration system with adaptive generation of instructions

Mixed Reality (MR) remote collaboration supports a shared workspace and various non-verbal communication cues that enable remote experts to express ideas in an immersive Virtual Reality (VR) space to guide local workers in industrial physical tasks by creating Augmented Reality (AR) instructions, such as remote assembly/disassembly guidance, emergency maintenance, and training. However, due to the information non-symmetrical caused by geographical distribution, it can be challenging and high-workload for remote experts to create clear and detailed AR instructions for local workers to match spatial relationships with the local workspace, while comprehensively considering the situation information (e.g., task-related objects and partner’s environment) in complex industrial physical tasks. Advances in ambient intelligence bring ideas for building a new remote collaboration framework that supports the adaptive generation of instructions. In the study, we developed a novel MR remote collaboration prototype system, which can automatically and adaptively generate clear, detailed, and standardized AR instructions based on remote experts’ simple and intuitive interactions and local contextual information to assist remote experts in guiding local workers to complete industrial physical tasks, especially industrial assembly tasks. The effect of two interfaces on industrial assembly tasks involving relatively complex process information and environments was explored in a user study: one was a baseline solution that uses a typical MR remote collaboration interface to support remote experts to create instructions (RECI) using popular non-verbal communication cues in the shared local 3D stereoscopic scene, and the other was our novel interface that additionally supported the adaptive generation of instructions via context awareness (AGICA) on top of the typical interface. The result of the user study showed that AGICA significantly improves collaboration performance, reduces errors and workload, and enhances usability and user experience compared to RECI. This demonstrates that supporting the adaptive generation of instructions is a feasible way to enhance MR remote collaboration in industrial assembly tasks with relatively complex process information and environments. Our research findings have a certain guiding significance for the design of MR remote collaboration systems in such tasks.

Visualizing Hand Force With Wearable Muscle Sensing for Enhanced Mixed Reality Remote Collaboration.

In this paper, we present a prototype system for sharing a user's hand force in mixed reality (MR) remote collaboration on physical tasks, where hand force is estimated using wearable surface electromyography (sEMG) sensor. In a remote collaboration between a worker and an expert, hand activity plays a crucial role. However, the force exerted by the worker's hand has not been extensively investigated. Our sEMG-based system reliably captures the worker's hand force during physical tasks and conveys this information to the expert through hand force visualization, overlaid on the worker's view or on the worker's avatar. A user study was conducted to evaluate the impact of visualizing a worker's hand force on collaboration, employing three distinct visualization methods across two view modes. Our findings demonstrate that sensing and sharing hand force in MR remote collaboration improves the expert's awareness of the worker's task, significantly enhances the expert's perception of the collaborator's hand force and the weight of the interacting object, and promotes a heightened sense of social presence for the expert. Based on the findings, we provide design implications for future mixed reality remote collaboration systems that incorporate hand force sensing and visualization.

Effects of Avatar Transparency on Social Presence in Task-centric Mixed Reality Remote Collaboration.

Despite the importance of avatar representation on user experience for Mixed Reality (MR) remote collaboration involving various device environments and large amounts of task-related information, studies on how controlling visual parameters for avatars can benefit users in such situations have been scarce. Thus, we conducted a user study comparing the effects of three avatars with different transparency levels (Nontransparent, Semi-transparent, and Near-transparent) on social presence for users in Augmented Reality (AR) and Virtual Reality (VR) during task-centric MR remote collaboration. Results show that avatars with a strong visual presence are not required in situations where accomplishing the collaborative task is prioritized over social interaction. However, AR users preferred more vivid avatars than VR users. Based on our findings, we suggest guidelines on how different levels of avatar transparency should be applied based on the context of the task and device type for MR remote collaboration.

Multi-Scale Interactions Between Augmented Reality and Virtual Reality Users in Mixed Reality Environment

Multi-scale interaction in Mixed Reality (MR) allows the avatars to shrink down themselves into a small size or enlarge into giant size. Multi-scale in MR is important in some situations that required precise avatar’s measurements to interact and manipulate with the content. However, the correct or appropriate scaling for multi-scale still remain unsolved in MR, very less people work in Augmented Reality (AR). It is critical to align precisely the way humans perceive the virtual world with the ways perceiving the real world. Therefore, this paper aims to produce a correct multi-scale technique for VR and AR users in MR environment. This paper presents a preliminary study on the previous works multi-scale techniques for MR remote collaboration. Then it explains the development of a multi-scale interaction between VR and AR users for the MR environment. Finally, this paper shows the results and implement the technique into an application. The results used to measure how multi-scale interaction can be leveraged with MR technology. The remote MR application involves AR and VR users with different interaction techniques.

Using Virtual Replicas to Improve Mixed Reality Remote Collaboration.

In this paper, we explore how virtual replicas can enhance Mixed Reality (MR) remote collaboration with a 3D reconstruction of the task space. People in different locations may need to work together remotely on complicated tasks. For example, a local user could follow a remote expert's instructions to complete a physical task. However, it could be challenging for the local user to fully understand the remote expert's intentions without effective spatial referencing and action demonstration. In this research, we investigate how virtual replicas can work as a spatial communication cue to improve MR remote collaboration. This approach segments the foreground manipulable objects in the local environment and creates corresponding virtual replicas of physical task objects. The remote user can then manipulate these virtual replicas to explain the task and guide their partner. This enables the local user to rapidly and accurately understand the remote expert's intentions and instructions. Our user study with an object assembly task found that using virtual replica manipulation was more efficient than using 3D annotation drawing in an MR remote collaboration scenario. We report and discuss the findings and limitations of our system and study, and present directions for future research.

A novel MR remote collaboration system using 3D spatial area cue and visual notification

In remote collaboration for emergency maintenance or training, since local workers may lack relevant experience and knowledge, it is very important to support remote experts to annotate 3D spatial areas for local workers in their workspace, such as risky spatial areas that need to be avoided for safe operation when assembling replacement parts. Recently, many researchers have studied various visual communication cues for expressing spatial information in Mixed Reality (MR) remote collaboration systems for physical tasks. However, the expression and transmission of 3D spatial area information have not been well explored, especially in tasks that need to annotate specific spatial areas in 3D space (such as risky 3D spatial areas). In the study, we developed a novel MR remote collaboration system that supports remote experts using 3D spatial area cues and visual notification to create Augmented Reality (AR) instructions for local workers on the shared 3D stereoscopic scene of the local workspace in the Virtual Reality (VR) space. We performed a user study to explore the effect of three interfaces on remote collaborative physical assembly tasks in the workspace with risky 3D spatial areas: 3D sketch cue (3DS) which is the baseline solution, 3D sketch cue combines the novel 3D spatial area cue (3DSA), and 3D sketch cue combines 3D spatial area cue and visual notification (3DSAN). We found the 3DSAN can significantly improve collaboration efficiency, reduce operation errors, and improve the usability and collaboration experience (especially the level of awareness), among industrial remote collaborative physical tasks requiring 3D spatial area reference.

3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration

As Virtual Reality(VR), Augmented Reality(AR), Mixed Reality(MR) technology becomes more accessible, it is important to explore VR/AR/MR technologies that can be used for remote collaboration on physical tasks. Previous research has shown that gesture-based interaction is intuitive and expressive for remote collaboration, and using 3D CAD models can provide clear instructions for assembly tasks. In this paper, therefore, we describe a new MR remote collaboration system which combines the use of gesture and CAD models in a complementary manner. The prototype system enables a remote expert in VR to provide instructions based on 3D gesture and CAD models (3DGAM) for a local worker who uses AR to see these instructions. Using this interface, we conducted a formal user study to explore the effect of sharing 3D gesture and CAD models in an assembly training task. We found that the combination of 3D gesture and CAD models can improve remote collaboration on an assembly task with respect to the performance time and user experience. Finally, we provide some conclusions and directions for future research.

Exploring interaction techniques for 360 panoramas inside a 3D reconstructed scene for mixed reality remote collaboration

Remote collaboration using mixed reality (MR) enables two separated workers to collaborate by sharing visual cues. A local worker can share his/her environment to the remote worker for a better contextual understanding. However, prior techniques were using either 360 video sharing or a complicated 3D reconstruction configuration. This limits the interactivity and practicality of the system. In this paper we show an interactive and easy-to-configure MR remote collaboration technique enabling a local worker to easily share his/her environment by integrating 360 panorama images into a low-cost 3D reconstructed scene as photo-bubbles and projective textures. This enables the remote worker to visit past scenes on either an immersive 360 panoramic scenery, or an interactive 3D environment. We developed a prototype and conducted a user study comparing the two modes of how 360 panorama images could be used in a remote collaboration system. Results suggested that both photo-bubbles and projective textures can provide high social presence, co-presence and low cognitive load for solving tasks while each have its advantage and limitations. For example, photo-bubbles are good for a quick navigation inside the 3D environment without depth perception while projective textures are good for spatial understanding but require physical efforts.

The combination of visual communication cues in mixed reality remote collaboration

Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when a dependent view and independent view are supported respectively. In the first user study with the dependent view, the results showed that the hand gesture cue was the main visual communication cue and adding sketch cue to the hand gesture cue helped participants complete the task faster. In the second study with the independent view, the results showed that the hand gesture had an issue of local worker understanding remote expert’s hand gesture cue and the main visual communication cue was the pointer cue with fast completion time and high level of co-presence.

The effects of spatial auditory and visual cues on mixed reality remote collaboration

Collaborative Mixed Reality (MR) technologies enable remote people to work together by sharing communication cues intrinsic to face-to-face conversations, such as eye gaze and hand gestures. While the role of visual cues has been investigated in many collaborative MR systems, the use of spatial auditory cues remains underexplored. In this paper, we present an MR remote collaboration system that shares both spatial auditory and visual cues between collaborators to help them complete a search task. Through two user studies in a large office, we found that compared to non-spatialized audio, the spatialized remote expert’s voice and auditory beacons enabled local workers to find small occluded objects with significantly stronger spatial perception. We also found that while the spatial auditory cues could indicate the spatial layout and a general direction to search for the target object, visual head frustum and hand gestures intuitively demonstrated the remote expert’s movements and the position of the target. Integrating visual cues (especially the head frustum) with the spatial auditory cues significantly improved the local worker’s task performance, social presence, and spatial perception of the environment.

A gesture- and head-based multimodal interaction platform for MR remote collaboration

In this paper, we present a projector-based mixed reality (MR) remote collaborative system which enables remote users to collaboratively work on a physical task using gesture and head pointing (GHP). Using this platform, we studied the effects of GHP in a typical manufacturing use case. Our system supports natural and intuitive multimodal interaction based on GHP, and it can project the remote user’s GHP into the local environment to enhance remote collaboration. Our prototype system was compared with an augmented reality (AR) condition (ANNOTATION), which is the most popular method currently for AR/MR remote collaboration. We found a significant difference between the ANNOTATION and GHP conditions in terms of performance. The GHP system significantly improved the collaborative experience (e.g., awareness of the user’s attention), empathy (e.g., co-presence), and remote interaction. Moreover, we discuss the implications of this research and directions for future research.

Superman vs Giant: A Study on Spatial Perception for a Multi-Scale Mixed Reality Flying Telepresence Interface.

The advancements in Mixed Reality (MR), Unmanned Aerial Vehicle, and multi-scale collaborative virtual environments have led to new interface opportunities for remote collaboration. This paper explores a novel concept of flying telepresence for multi-scale mixed reality remote collaboration. This work could enable remote collaboration at a larger scale such as building construction. We conducted a user study with three experiments. The first experiment compared two interfaces, static and dynamic IPD, on simulator sickness and body size perception. The second experiment tested the user perception of a virtual object size under three levels of IPD and movement gain manipulation with a fixed eye height in a virtual environment having reduced or rich visual cues. Our last experiment investigated the participant's body size perception for two levels of manipulation of the IPDs and heights using stereo video footage to simulate a flying telepresence experience. The studies found that manipulating IPDs and eye height influenced the user's size perception. We present our findings and share the recommendations for designing a multi-scale MR flying telepresence interface.

遠隔ユーザの注意を作業物体上に反映するMR遠隔コラボレーション(「仮想都市とVR」特集)

遠隔ユーザの注意を作業物体上に反映するMR遠隔コラボレーション(「仮想都市とVR」特集)