Immersive Telepresence Research Articles

Immersive Mobile Telepresence Systems: A Systematic Literature Review

Telepresence can be defined as a system that provides remote collaboration between people in different locations, creating the feeling that both users share in fact the environment. The advances in communication, media and internet, has made possible the popularization of these systems. Smartphones have become increasingly powerful in processing, less expensive and more widespread. A single device combines various sensors, one or more cameras and internet connection, thus a potential hardware for telepresence applications. The main objective of this paper is to present a Systematic Literature Review to identify the main characteristics of immersive telepresence systems designed for mobile environment and to analyze research opportunities that can be further exploited or optimized. This research revealed that the development of immersive telepresence systems for mobile devices has increased in recent years, but is not yet widespread.

Flexible Disaster Response of Tomorrow: Final Presentation and Evaluation of the CENTAURO System

Mobile manipulation robots have great potential for roles in support of rescuers on disaster-response missions. Robots can operate in places too dangerous for humans and therefore can assist in accomplishing hazardous tasks while their human operators work at a safe distance. We developed a disaster-response system that consists of the highly flexible Centauro robot and suitable control interfaces, including an immersive telepresence suit and support-operator controls offering different levels of autonomy.

There is a world elsewhere: rehearsing and training through immersive telepresence

The article will examine how telepresence technologies are repurposed to enable performers to rehearse and train over distances. The discussion will revolve around a variety of telepresence technologies/web-based applications repurposed to investigate actor training, rehearsal, education and performance. In this project, Immersive Telepresence in Theatre, two identical spaces, linked by H.323 videoconferencing technology, were created in Tampere and Coventry, each with rear projection screens giving the participants the impression of a shared space. The project used existing pedagogical practices but re-applied them to a digital setting, with the teaching and rehearsal process only slightly modified to account for the technological aspects of working in this manner. As well as the traditional rehearsal process there is also the social aspect which builds a group dynamic amongst the performers. For this aspect of the process, a range of supporting applications were used – Facebook for scheduling and rapid sharing of visual materials from and the web conferencing app, Adobe Connect, was used to provide individualised rehearsal rooms for the participants as well as break-out social spaces. When training in a digital environment, the lack of immediate touch must be replaced by other sensory means. The use of the screen meant that actors had to take a more experimental approach to rehearsal. This process redefines how participants use the space, as proxemics are replaced with visual representation on the screen. Here ‘play’ and the participants’ own will to investigate the non-traditional possibilities of this process became tantamount.

Immersive Telepresence and Remote Collaboration using Mobile and Wearable Devices.

The mobility and ubiquity of mobile head-mounted displays make them a promising platform for telepresence research as they allow for spontaneous and remote use cases not possible with stationary hardware. In this work we present a system that provides immersive telepresence and remote collaboration on mobile and wearable devices by building a live spherical panoramic representation of a user's environment that can be viewed in real time by a remote user who can independently choose the viewing direction. The remote user can then interact with this environment as if they were actually there through intuitive gesture-based interaction. Each user can obtain independent views within this environment by rotating their device, and their current field of view is shared to allow for simple coordination of viewpoints. We present several different approaches to create this shared live environment and discuss their implementation details, individual challenges, and performance on modern mobile hardware; by doing so we provide key insights into the design and implementation of next generation mobile telepresence systems, guiding future research in this domain. The results of a preliminary user study confirm the ability of our system to induce the desired sense of presence in its users.

Unequal Error Protection Aided Region of Interest Aware Wireless Panoramic Video

Panoramic video with its flawless immersive tele-presence is considered to be the near-future video format of choice since they carry 360 degree coverage of the designated scenes. However, the viewers may focus their specific attention on perfectly lip-synchronized video as part of the panoramic video scene, hence only have a peripheral vision of the remaining parts of a frame. Therefore, it is intuitive to allocate stronger protection to the panoramic video region of interest. As a solution, we propose Region of Interest Aware Unequal Error Protection (ROI-UEP) for wireless transmission of high-efficiency video code (HEVC) sequences. Specifically, the ROI of a panoramic frame may be deemed to be within the 120° angular range of the viewing center, which can be estimated from the viewing trajectory of a head mounted display. Then, the most appropriate unequal forward error correction (FEC) coding rates will be found for the ROI signals by minimizing the expected video distortion. Moreover, the so-called weighted peak signal-to-noise ratio (WPSNR) is proposed for evaluating the quality of the reconstructed panoramic video, where the weights of pixels are taken into account for calculating the distortion caused by the related pixels. Our simulation results show that the ROI based equal error protection (ROI-EEP) scheme substantially outperforms the EEP by a WPSNR of more than 10 dB, while the ROI-UEP scheme further improves its ROI-EEP counterpart by a WPSNR of 9.4 dB at a channel E b /N 0 of 6 dB.

Distributed Consolidation of Highly Incomplete Dynamic Point Clouds Based on Rank Minimization

Recently, there has been increasing interest for easy and reliable generation of 3-D animated models facilitating several real-time applications (like immersive telepresence, motion capture, and gaming). In most of these applications, the reconstruction of soft body animations is based on time-varying point clouds, which are nonuniformly sampled and highly incomplete. To overcome these significantly challenging imperfections without any additional information, first we introduce a novel reconstruction technique based on rank minimization theory, which can result in a unique solution to the otherwise ill-posed problem. This technique is further extended to exploit the spatial coherence, which usually characterizes the soft-body animations. Based on the developed tools, we propose a distributed consolidation technique where the reconstruction is performed by working simultaneously on several groups of frames. To achieve this, we impose temporal coherence between successive frame clusters by constraining the rank minimization problem. We validate the proposed techniques via experimental evaluation under different configurations and animated models, where we show that the high-frequency details of the models can be adequately recovered from a highly incomplete geometry data set.

Fast Depth Video Compression for Mobile RGB-D Sensors

We propose a new method, called 3-D image warping-based depth video compression (IW-DVC), for fast and efficient compression of depth images captured by mobile RGB-D sensors. The emergence of low-cost RGB-D sensors has created opportunities to find new solutions for a number of computer vision and networked robotics problems, such as 3-D map building, immersive telepresence, or remote sensing. However, efficient transmission and storage of depth data still presents a challenging task to the research community in these applications. Image/video compression has been comprehensively studied and several methods have already been developed. However, these methods result in unacceptably suboptimal outcomes when applied to the depth images. We have designed the IW-DVC method to exploit the special properties of the depth data to achieve a high compression ratio while preserving the quality of the captured depth images. Our solution combines the egomotion estimation and 3-D image warping techniques and includes a lossless coding scheme that is capable of adapting to depth data with a high dynamic range. IW-DVC operates at a high speed, suitable for real-time applications, and is able to attain an enhanced motion compensation accuracy compared with the conventional approaches. In addition, it removes the existing redundant information between the depth frames to further increase compression efficiency. Our experiments show that IW-DVC attains a very high performance yielding significant compression ratios without sacrificing image quality.

ITEM: Immersive Telepresence for Entertainment and Meetings—A Practical Approach

This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive (TI) application scenarios. In addition, we also present a few key technologies, i.e., fast object-based video coding for real world data and spatialized audio capture and 3-D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g., a webcam or a color plus depth camera, an optional microphone array, a laptop/desktop connected to the Internet) for TI communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.

Estimating the Gaze of a Virtuality Human

The aim of our experiment is to determine if eye-gaze can be estimated from a virtuality human: to within the accuracies that underpin social interaction; and reliably across gaze poses and camera arrangements likely in every day settings. The scene is set by explaining why Immersive Virtuality Telepresence has the potential to meet the grand challenge of faithfully communicating both the appearance and the focus of attention of a remote human participant within a shared 3D computer-supported context. Within the experiment n=22 participants rotated static 3D virtuality humans, reconstructed from surround images, until they felt most looked at. The dependent variable was absolute angular error, which was compared to that underpinning social gaze behaviour in the natural world. Independent variables were 1) relative orientations of eye, head and body of captured subject; and 2) subset of cameras used to texture the form. Analysis looked for statistical and practical significance and qualitative corroborating evidence. The analysed results tell us much about the importance and detail of the relationship between gaze pose, method of video based reconstruction, and camera arrangement. They tell us that virtuality can reproduce gaze to an accuracy useful in social interaction, but with the adopted method of Video Based Reconstruction, this is highly dependent on combination of gaze pose and camera arrangement. This suggests changes in the VBR approach in order to allow more flexible camera arrangements. The work is of interest to those wanting to support expressive meetings that are both socially and spatially situated, and particular those using or building Immersive Virtuality Telepresence to accomplish this. It is also of relevance to the use of virtuality humans in applications ranging from the study of human interactions to gaming and the crossing of the stage line in films and TV.

Immersive Group-to-Group Telepresence

We present a novel immersive telepresence system that allows distributed groups of users to meet in a shared virtual 3D world. Our approach is based on two coupled projection-based multi-user setups, each providing multiple users with perspectively correct stereoscopic images. At each site the users and their local interaction space are continuously captured using a cluster of registered depth and color cameras. The captured 3D information is transferred to the respective other location, where the remote participants are virtually reconstructed. We explore the use of these virtual user representations in various interaction scenarios in which local and remote users are face-to-face, side-by-side or decoupled. Initial experiments with distributed user groups indicate the mutual understanding of pointing and tracing gestures independent of whether they were performed by local or remote participants. Our users were excited about the new possibilities of jointly exploring a virtual city, where they relied on a world-in-miniature metaphor for mutual awareness of their respective locations.

Validation of a Testbed as a Screening Tool for Telepresence Systems in Terms of Fine-Motor Performance

Telepresence systems have the potential to assume an important role in on-orbit servicing scenarios. Compared to manned missions, these systems offer a safer and less expensive way to operate in outer space. One of the main goals of telepresence research is to determine whether immersive telepresence systems are able to achieve a level of efficiency which is equal to that of astronauts in tasks that require fine motor movement and dexterity. In addressing this question, it is of relevance that astronauts' fine-motor skills are restricted by a range of factors including weightlessness and space suits. In order to compare in future studies the efficiency of a telepresence system and suited astronauts in performing fine-motor tasks, a testbed was developed with typical on-orbit servicing tasks which were designed to guarantee maximum external validity. An experimental study showed that the testbed is a valid measure of fine-motor skills. Two follow-up studies examined the potentially debilitating influence of a range of factors on fine-motor performance in extravehicular activities, such as a restricted field of view, a destabilized body posture, restricted mobility, and reduced haptic feedback. The results of these studies offer a basis for comparisons between the performance of common extravehicular activities and telepresence systems on fine-motor tasks.

DESIGN REVIEW IN A DISTRIBUTED COLLABORATIVE VIRTUAL ENVIRONMENT

Virtual Reality leads Digital Product Development technology to a more effective and convenient way by presenting the product model in an immersive real 3D environment. Distributed VR even enables partners to discuss the design of the product independent of their locations. This paper introduces our work with a Digital Product Development System with Virtually Cooperating Teams Using High Bandwidth Networks — ProViT, which aims at building up immersive telepresence systems for design review across high-bandwidth networks. Video and audio streaming are integrated in the virtual environment to support immersive teleconference aspects of the distributed design review sessions. A network management system has been developed for the supervision, maintenance and processing of distributed setups, and network security.

Stereo-Based Environment Scanning for Immersive Telepresence

The processing power and network bandwidth required for true immersive telepresence applications are only now beginning to be available. We draw from our experience developing stereo based tele-immersion prototypes to present the main issues arising when building these systems. Tele-immersion is a new medium that enables a user to share a virtual space with remote participants. The user is immersed in a rendered three-dimensional (3-D) world that is transmitted from a remote site. To acquire this 3-D description, we apply binocular and trinocular stereo techniques which provide a view-independent scene description. Slow processing cycles or long network latencies interfere with the users' ability to communicate, so the dense stereo range data must be computed and transmitted at high frame rates. Moreover, reconstructed 3-D views of the remote scene must be as accurate as possible to achieve a sense of presence. We address both issues of speed and accuracy using a variety of techniques including the power of supercomputing clusters and a method for combining motion and stereo in order to increase speed and robustness. We present the latest prototype acquiring a room-size environment in real time using a supercomputing cluster, and we discuss its strengths and current weaknesses.

Mechatronics applied to auditory localisation for telepresence

This paper describes the importance of the sense of hearing for immersive telepresence systems, and a fundamental approach to designing and implementing a binaural sensor system. This approach relies on knowledge of how the human auditory localisation mechanisms work to locate sounds in space, in order that a synthetic system can be derived. The most important of these mechanisms are explained. The Strathclyde auditory system is then described including ideas drawn from the literature to define a binaural approach. This novel system consists of a set of artificial pinnae and microphones mounted on a binocular stereo sensor platform capable of pan, tilt and roll which is slaved to a head tracking system. A set of preliminary experiments is described which test the performance of the system in practice, as a basis for further development.

Immersive Telepresence

Immersive Telepresence