Immersive Telepresence Research Articles

The $10 Million ANA Avatar XPRIZE Competition: How It Advanced Immersive Telepresence Systems

The $10 Million ANA Avatar XPRIZE Competition: How It Advanced Immersive Telepresence Systems

NimbRo Wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned

AbstractRobotic avatar systems can enable immersive telepresence with locomotion, manipulation, and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic upper body with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing omnidirectional locomotion on flat floors, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR display worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying an animated image of the operator’s face, enabling direct interaction with remote persons. Our system won the $10 M ANA Avatar XPRIZE competition, which challenged teams to develop intuitive and immersive avatar systems that could be operated by briefly trained judges. We analyze our successful participation in the semifinals and finals and provide insight into our operator training and lessons learned. In addition, we evaluate our system in a user study that demonstrates its intuitive and easy usability.

Real-Time 3D Reconstruction Pipeline for Room-Scale, Immersive, Medical Teleconsultation

Medical teleconsultation was among the initial use cases for early telepresence research projects since medical treatment often requires timely intervention by highly specialized experts. When remote medical experts support interventions, a holistic view of the surgical site can increase situation awareness and improve team communication. A possible solution is the concept of immersive telepresence, where remote users virtually join the operating theater that is transmitted based on a real-time reconstruction of the local site. Enabled by the availability of RGB-D sensors and sufficient computing capability, it becomes possible to capture such a site in real time using multiple stationary sensors. The 3D reconstruction and simplification of textured surface meshes from the point clouds of a dynamic scene in real time is challenging and becomes infeasible for increasing capture volumes. This work presents a tightly integrated, stateless 3D reconstruction pipeline for dynamic, room-scale environments that generates simplified surface meshes from multiple RGB-D sensors in real time. Our algorithm operates directly on the fused, voxelized point cloud instead of populating signed-distance volumes per frame and using a marching cube variant for surface reconstruction. We extend the formulation of the dual contouring algorithm to work for point cloud data stored in an octree and interleave a vertex-clustering-based simplification before extracting the surface geometry. Our 3D reconstruction pipeline can perform a live reconstruction of six incoming depth videos at their native frame rate of 30 frames per second, enabling the reconstruction of smooth movement. Arbitrarily complex scene changes are possible since we do not store persistent information between frames. In terms of mesh quality and hole filling, our method falls between the direct mesh reconstruction and expensive global fitting of implicit functions.

Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

Novel view synthesis has attracted tremendous research attention recently for its applications in virtual reality and immersive telepresence. Rendering a locally immersive light field (LF) based on arbitrary large baseline RGB references is a challenging problem that lacks efficient solutions with existing novel view synthesis techniques. In this work, we aim at truthfully rendering local immersive novel views/LF images based on large baseline LF captures and a single RGB image in the target view. To fully explore the precious information from source LF captures, we propose a novel occlusion-aware source sampler (OSS) module which efficiently transfers the pixels of source views to the target view’s frustum in an occlusion-aware manner. An attention-based deep visual fusion module is proposed to fuse the revealed occluded background content with a preliminary LF into a final refined LF. The proposed source sampling and fusion mechanism not only helps to provide information for occluded regions from varying observation angles, but also proves to be able to effectively enhance the visual rendering quality. Experimental results show that our proposed method is able to render high-quality LF images/novel views with sparse RGB references and outperforms state-of-the-art LF rendering and novel view synthesis methods.

Immersive robotic telepresence system to support a person with intellectual and motor disabilities perform a daily task: a case study

PurposeIn this proof-of-concept study, the authors assessed the feasibility of using a humanoid robot controlled remotely via an immersive telepresence system to support a person with intellectual and motor disabilities performing a daily task (i.e. setting a table for lunch).Design/methodology/approachThe system involved a head-mounted display and two joysticks. A teleoperator was able to see through the video cameras of the robot and deliver the instructions verbally to the participant located in a different room. To assess the system, a baseline phase (A) was followed by an intervention (i.e. tele-operated support) phase (B) and a return to a baseline phase (A).FindingsData showed a marked increase in the average frequency of task steps correctly performed from baseline (M = 15%) to intervention (M = 93%). Accuracy reached 100% in the return to baseline.Originality/valueThese preliminary findings, along with qualitative feedback from users, suggest that an immersive telepresence system may be used to provide remote support to people with intellectual and motor disabilities.

Augmenting Immersive Telepresence Experience with a Virtual Body.

We propose augmenting immersive telepresence by adding a virtual body, representing the user's own arm motions, as realized through a head-mounted display and a 360-degree camera. Previous research has shown the effectiveness of having a virtual body in simulated environments; however, research on whether seeing one's own virtual arms increases presence or preference for the user in an immersive telepresence setup is limited. We conducted a study where a host introduced a research lab while participants wore a head-mounted display which allowed them to be telepresent at the host's physical location via a 360-degree camera, either with or without a virtual body. We first conducted a pilot study of 20 participants, followed by a pre-registered 62 participant confirmatory study. Whereas the pilot study showed greater presence and preference when the virtual body was present, the confirmatory study failed to replicate these results, with only behavioral measures suggesting an increase in presence. After analyzing the qualitative data and modeling interactions, we suspect that the quality and style of the virtual arms, and the contrast between animation and video, led to individual differences in reactions to the virtual body which subsequently moderated feelings of presence.

Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering.

Image-based geometric modeling and novel view synthesis based on sparse large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, the fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules (i.e., the attention-guided multi-scale residual fusion module, and the disparity field-guided deep re-regularization module), which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs, have been proposed. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis.

Deep Point Set Resampling via Gradient Fields.

3D point clouds acquired by scanning real-world objects or scenes have found a wide range of applications including immersive telepresence, autonomous driving, surveillance, etc. They are often perturbed by noise or suffer from low density, which obstructs downstream tasks such as surface reconstruction and understanding. In this paper, we propose a novel paradigm of point set resampling for restoration, which learns continuous gradient fields of point clouds that converge points towards the underlying surface. In particular, we represent a point cloud via its gradient field-the gradient of the log-probability density function, and enforce the gradient field to be continuous, thus guaranteeing the continuity of the model for solvable optimization. Based on the continuous gradient fields estimated via a proposed neural network, resampling a point cloud amounts to performing gradient-based Markov Chain Monte Carlo (MCMC) on the input noisy or sparse point cloud. Further, we propose to introduce regularization into the gradient-based MCMC during point cloud restoration, which essentially refines the intermediate resampled point cloud iteratively and accommodates various priors in the resampling process. Extensive experimental results demonstrate that the proposed point set resampling achieves the state-of-the-art performance in representative restoration tasks including point cloud denoising and upsampling.

Graph Signal Processing for Geometric Data and Beyond: Theory and Applications

Geometric data acquired from real-world scenes, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , 2D depth images, 3D point clouds, and 4D dynamic point clouds, have found a wide range of applications including immersive telepresence, autonomous driving, surveillance, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc</i> . Due to irregular sampling patterns of most geometric data, traditional image/video processing methodologies are limited, while Graph Signal Processing (GSP)—a fast-developing field in the signal processing community—enables processing signals that reside on irregular domains and plays a critical role in numerous applications of geometric data from low-level processing to high-level analysis. To further advance the research in this field, we provide the first timely and comprehensive overview of GSP methodologies for geometric data in a unified manner by bridging the connections between geometric data and graphs, among the various geometric data modalities, and with spectral/nodal graph filtering techniques. We also discuss the recently developed Graph Neural Networks (GNNs) and interpret the operation of these networks from the perspective of GSP. We conclude with a brief discussion of open problems and challenges.

Development of an immersive SLAM-based VR system for teleoperation of a mobile manipulator in an unknown environment

Operating a remote robot in an unknown environment is a challenge. Usually, a camera is attached to the robot to capture the images of the remote scenes. However, it is difficult to perceive a 3D environment using a 2D display. In this study, a real-time simultaneous localization and mapping (SLAM)-based virtual reality (VR) 3D human-machine interaction system is developed to provide users with immersive telepresence to better operate a remote mobile manipulator in an unknown environment. An RGB-depth camera is mounted on the mobile platform to scan and reconstruct the remote environment in real time. A head mounted display is used to display the 3D reconstructed environment, with a virtual copy of the actual robot in it, to provide users the feeling of immersion at the remote site. Visual aids are added in the 3D reconstructed environment to guide users with the direction and distance information. Users can control the remote robot by manipulating the virtual copy of the actual robot in the VR environment, without being constrained in the robot first person view. The system is compared with a video-based teleoperation and a direct on-site operation. Experimental results show that the grasping error and placing threshold of the SLAM-based VR teleoperation are 0.5 cm and 1.5 cm, respectively. The average error of the reconstructed environment is less than 4%. Within the accuracy of the reconstructed environment, the SLAM-based VR teleoperation has a higher grasping success rate and lower operation time, compared to the video-based teleoperation. This study shows a promising application of immersive SLAM-based VR in interactive telerobotic operations in a remote unknown environment.

A Multi-Site Evaluation of Observer Roles and Telepresence Technology in Simulation with Prelicensure Nursing Students

BackgroundMany learners are observers and do not directly participate in the care for a simulated patient. Immersive telepresence technology makes it possible to don virtual reality goggles, giving learners their own vantage point, making them feel present in the room. MethodsA multisite quasi-experimental design was used to explore differences in perceived levels of presence and knowledge outcomes among three common roles in simulation and an immersive telepresence role. ResultsLearners in immersive telepresence observer roles had significantly higher presence scores than learners who observed via flat screen and projector in an audio-visual room. Immersive telepresence observer roles and active participant roles had similar scores of perceived presence. There were no significant differences in knowledge outcomes among learner roles. ConclusionsImmersive telepresence technology offers an ability to scale simulations for larger groups and demonstrates higher levels of perceived learner engagement and presence than other observer roles.

Dynamic Point Cloud Denoising via Manifold-to-Manifold Distance.

3D dynamic point clouds provide a natural discrete representation of real-world objects or scenes in motion, with a wide range of applications in immersive telepresence, autonomous driving, surveillance, etc. Nevertheless, dynamic point clouds are often perturbed by noise due to hardware, software or other causes. While a plethora of methods have been proposed for static point cloud denoising, few efforts are made for the denoising of dynamic point clouds, which is quite challenging due to the irregular sampling patterns both spatially and temporally. In this paper, we represent dynamic point clouds naturally on spatial-temporal graphs, and exploit the temporal consistency with respect to the underlying surface (manifold). In particular, we define a manifold-to-manifold distance and its discrete counterpart on graphs to measure the variation-based intrinsic distance between surface patches in the temporal domain, provided that graph operators are discrete counterparts of functionals on Riemannian manifolds. Then, we construct the spatial-temporal graph connectivity between corresponding surface patches based on the temporal distance and between points in adjacent patches in the spatial domain. Leveraging the initial graph representation, we formulate dynamic point cloud denoising as the joint optimization of the desired point cloud and underlying graph representation, regularized by both spatial smoothness and temporal consistency. We reformulate the optimization and present an efficient algorithm. Experimental results show that the proposed method significantly outperforms independent denoising of each frame from state-of-the-art static point cloud denoising approaches, on both Gaussian noise and simulated LiDAR noise.

Immersive telepresence on the operation of unmanned vehicles

Unmanned ground vehicles are usually deployed in situations, where it is too dangerous or not feasible to have an operator onboard. One challenge faced when such vehicles are teleoperated is maintaining a high situational awareness, due to aspects such as limitation of cameras, characteristics of network transmission, and the lack of other sensory information, such as sounds and vibrations. Situation awareness refers to the understanding of the information, events, and actions that will impact the execution and the objectives of the tasks at the current and near future of the operation of the vehicle. This work investigates how the simultaneous use of immersive telepresence and mixed reality could impact the situation awareness of the operator and the navigation performance. A user study was performed to compare our proposed approach with a traditional unmanned vehicle control station. Quantitative data obtained from the vehicle’s behavior and the situation awareness global assessment technique were used to analyze such impacts. Results provide evidence that our approach is relevant when the task requires a detailed observation of the surroundings, leading to higher situation awareness and navigation performance.

Dynamic Point Cloud Inpainting via Spatial-Temporal Graph Learning

The maturity of depth sensors and laser scanning techniques has enabled the convenient acquisition of 3D dynamic point clouds—one natural representation of 3D objects/scenes in motion, leading to a wide range of applications such as immersive tele-presence, autonomous driving, augmented and virtual reality. Nevertheless, dynamic point clouds usually exhibit holes of missing data, thus inpainting is crucial to the subsequent rendering or downstream understanding tasks. Dynamic point cloud inpainting has been largely overlooked so far, which is also quite challenging due to the irregular sampling patterns both in the spatial domain and temporal domain. To this end, we propose an efficient dynamic point cloud inpainting method based on a learnable spatial-temporal graph representation, exploiting both the second-order inter-frame coherence and the intra-frame self-similarity. The key is the second-order inter-frame coherence that enforces the consistent flow in 3D motion over time, for which we search the temporal correspondence in consecutive frames for the same underlying surface by the point-to-plane distance and represent the correlation between them via temporal edge weights in the graph. Based on the second-order inter-frame coherence and intra-frame self-similarity, we formulate dynamic point cloud inpainting as a joint optimization problem of the desired point cloud and underlying spatial-temporal graph, which is regularized by consistency in the temporal edge weights and smoothness in the spatial domain. We analyze and reformulate the optimization, leading to an efficient alternating minimization algorithm. Experimental results show that the proposed approach outperforms several competing methods significantly, both on synthetic holes and real holes.

6G Vision, Value, Use Cases and Technologies From European 6G Flagship Project Hexa-X

While 5G is being deployed and the economy and society begin to reap the associated benefits, the research and development community starts to focus on the next, 6th Generation (6G) of wireless communications. Although there are papers available in the literature on visions, requirements and technical enablers for 6G from various academic perspectives, there is a lack of joint industry and academic work towards 6G. In this paper a consolidated view on vision, values, use cases and key enabling technologies from leading industry stakeholders and academia is presented. The authors represent the mobile communications ecosystem with competences spanning hardware, link layer and networking aspects, as well as standardization and regulation. The second contribution of the paper is revisiting and analyzing the key concurrent initiatives on 6G. A third contribution of the paper is the identification and justification of six key 6G research challenges: (i) “connecting”, in the sense of empowering, exploiting and governing, intelligence; (ii) realizing a network of networks, i.e., leveraging on existing networks and investments, while reinventing roles and protocols where needed; (iii) delivering extreme experiences, when/where needed; (iv) (environmental, economic, social) sustainability to address the major challenges of current societies; (v) trustworthiness as an ingrained fundamental design principle; (vi) supporting cost-effective global service coverage. A fourth contribution is a comprehensive specification of a concrete first-set of industry and academia jointly defined use cases for 6G, e.g., massive twinning, cooperative robots, immersive telepresence, and others. Finally, the anticipated evolutions in the radio, network and management/orchestration domains are discussed.

PECULIARITIES OF INVESTIGATION OF CRIMINAL CASES ASSOCIATED WITH ARSONS

The article discusses some of the features of the investigation of criminal cases involving arson. Recommendations are given for the officers of the investigative-operational groups when planning further actions at the initial stage of the investigation of arson. The questions about the influence on the detection rate of the promptness of arrival to the place of the fire of the investigator (interrogator), about the features of collecting information at the place of the fire are touched upon. Obtaining information on the basis of special knowledge, allowing to formulate a search plan for the alleged offender and his criminological personality portrait. With the development of new technologies, for the efficiency of obtaining information from a specialist (expert) in the investigation of arson, it is necessary to use the technology of immersive telepresence (video presence). This allows for a better inspection of the crime scene in the absence of an expert (specialist) directly on the spot. The use of electronic databases and the use of modern means of communication also contribute to solving crimes related to arson hot on the trail.

A Robust Real-Time 3D Reconstruction Method for Mixed Reality Telepresence

Mixed Reality (MR) is a technology which enable to bring a virtual element into the real-world environment. MR intends to improve reality on the virtual world immerse onto real-world space. Occasionally the MR has been improved as the display technologies advanced progressively. In MR collaborative interface context, the local and remote user work together on collaborative task while sense the immersive environment in the cooperative application. User telepresence is an immersive telepresence, where the reconstruction of a human appears in a real-life. Up till now, producing full telepresence of the life-size human body may require a high transmission bandwidth of the internet. Therefore, this paper explores on a robust real-time 3D reconstruction method for MR telepresence. This paper discusses the previous works on the reconstruction method of a full-body human and the existing research works that have proposed the reconstruction methods for telepresence. Besides the 3D reconstruction method, this paper also enlightens our recent finding on the MR framework to transport a full-body human from a local location to a remote location. The MR telepresence will be discussed, as well as the robust 3D reconstruction method which has been implemented with user telepresence feature where the user experiences an accurate 3D representation of a remote person. The paper ends with the discussion and results, MR telepresence with robust 3D reconstruction method to execute user telepresence.

Output-Sensitive Avatar Representations for Immersive Telepresence.

In this article, we propose a system design and implementation for output-sensitive reconstruction, transmission and rendering of 3D video avatars in distributed virtual environments. In our immersive telepresence system, users are captured by multiple RGBD sensors connected to a server that performs geometry reconstruction based on viewing feedback from remote telepresence parties. This feedback and reconstruction loop enables visibility-aware level-of-detail reconstruction of video avatars regarding geometry and texture data, and considers individual and groups of collocated users. Our evaluation reveals that our approach leads to a significant reduction of reconstruction times, network bandwidth requirements and round-trip times as well as rendering costs in many situations.

Predictive Generalized Graph Fourier Transform for Attribute Compression of Dynamic Point Clouds

As 3D scanning devices and depth sensors advance, dynamic point clouds have attracted increasing attention as a format for 3D objects in motion, with applications in various fields such as immersive telepresence, navigation for autonomous driving and gaming. Nevertheless, the tremendous amount of data in dynamic point clouds significantly burden transmission and storage. To this end, we propose a complete compression framework for attributes of 3D dynamic point clouds, focusing on optimal inter-coding. Firstly, we derive the optimal inter-prediction and predictive transform coding assuming the Gaussian Markov Random Field model with respect to a spatio-temporal graph underlying the attributes of dynamic point clouds. The optimal predictive transform proves to be the Generalized Graph Fourier Transform in terms of spatio-temporal decorrelation. Secondly, we propose refined motion estimation via efficient registration prior to inter-prediction, which searches the temporal correspondence between adjacent frames of irregular point clouds. Finally, we present a complete framework based on the optimal inter-coding and our previously proposed intra-coding, where we determine the optimal coding mode from rate-distortion optimization with the proposed offline-trained λ-Q model. Experimental results show that we achieve around 17% bit rate reduction on average over competitive dynamic point cloud compression methods.

On the Assessment of Functional Connectivity in an Immersive Brain-Computer Interface During Motor Imagery.

New trends on brain-computer interface (BCI) design are aiming to combine this technology with immersive virtual reality in order to provide a sense of realism to its users. In this study, we propose an experimental BCI to control an immersive telepresence system using motor imagery (MI). The system is immersive in the sense that the users can control the movement of a NAO humanoid robot in a first person perspective (1PP), i.e., as if the movement of the robot was his/her own. We analyze functional brain connectivity between 1PP and 3PP during the control of our BCI using graph theory properties such as degree, betweenness centrality, and efficiency. Changes in these metrics are obtained for the case of the 1PP, as well as for the traditional third person perspective (3PP) in which the user can see the movement of the robot as feedback. As proof-of-concept, electroencephalography (EEG) signals were recorded from two subjects while they performed MI to control the movement of the robot. The graph theoretical analysis was applied to the binary directed networks obtained through the partial directed coherence (PDC). In our preliminary assessment we found that the efficiency in the α brain rhythm is greater in 1PP condition in comparison to the 3PP at the prefrontal cortex. Also, a stronger influence of signals measured at EEG channel C3 (primary motor cortex) to other regions was found in 1PP condition. Furthermore, our preliminary results seem to indicate that α and β brain rhythms have a high indegree at prefrontal cortex in 1PP condition, and this could be possibly related to the experience of sense of agency. Therefore, using the PDC combined with graph theory while controlling a telepresence robot in an immersive system may contribute to understand the organization and behavior of brain networks in these environments.