Telepresence Applications Research Articles

Investigation of human interaction with an augmented remote operating system for scaling in mining applications

Thanks to the advent of telepresence applications, we can remotely take control and operate industrial machinery. Teleoperation removes operators from hazardous workplaces such as mining and plays an essential role in the safety of workers. In addition, augmented telepresence can introduce information that helps the user understand the remote scene. However, remote operation presents challenges since the information received is more limited than what could be perceived by being physically present, such as accurately judging depth. This study investigates how well operators interact with an augmented remote operation scaling system (AROSS) in a mining context when different computer-generated visual interfaces are provided. The system can achieve five visual interfaces: Disocclusion Augmentation view using selective content removal, Novel Perspective view generation, Lidar view, Right (Original) view, and Left (Original) view. We performed two experiments in a mine-like laboratory to analyze human interaction with the designed prototype by applying a mixed research methodology that used questionnaires, interviews, and observations. This mixed methodology consisted of quality of experience methods to discover the users’ requirements from a technological standpoint and user experience methods (i.e., user-centric approaches). We investigated 10 and 11 users’ interactions in two experimental studies. The first experiment focused on identifying small patterns (e.g., cracks in the mine wall), and the second focused on depth and three-dimensional understanding. We considered the first experiment a feasibility test to understand how to conduct the second experiment. Therefore, we designed the second test to assess the technical readiness of AROSS from the users’ perspective. The overall conclusion yields a comprehensive understanding of users’ perceptions and experiences. The quality of experience results favored Left and Right (Original) views for remote control, indicating a preference among remote operators using natural (Original) views due to their facilitation of environmental comprehension. User experience analysis revealed the reason why other views were less favored and what their potential benefits are. Specifically, Novel Perspective and Lidar views were found helpful for depth perception, and Disocclusion Augmentation view functionality could be enhanced if robot arm position tracking is enabled. These insights inform design recommendations, emphasizing the value of incorporating Disocclusion Augmentation and Novel Perspective views and suggesting improvements to enhance system usability.

N-DEPTH: Neural Depth Encoding for Compression-Resilient 3D Video Streaming

Recent advancements in 3D data capture have enabled the real-time acquisition of high-resolution 3D range data, even in mobile devices. However, this type of high bit-depth data remains difficult to efficiently transmit over a standard broadband connection. The most successful techniques for tackling this data problem thus far have been image-based depth encoding schemes that leverage modern image and video codecs. To our knowledge, no published work has directly optimized the end-to-end losses of a depth encoding scheme sandwiched around a lossy image compression codec. We present N-DEPTH, a compression-resilient neural depth encoding method that leverages deep learning to efficiently encode depth maps into 24-bit RGB representations that minimize end-to-end depth reconstruction errors when compressed with JPEG. N-DEPTH’s learned robustness to lossy compression expands to video codecs as well. Compared to an existing state-of-the-art encoding method, N-DEPTH achieves smaller file sizes and lower errors across a large range of compression qualities, in both image (JPEG) and video (H.264) formats. For example, reconstructions from N-DEPTH encodings stored with JPEG had dramatically lower error while still offering 29.8%-smaller file sizes. When H.264 video was used to target a 10 Mbps bit rate, N-DEPTH reconstructions had 85.1%-lower root mean square error (RMSE) and 15.3%-lower mean absolute error (MAE). Overall, our method offers an efficient and robust solution for emerging 3D streaming and 3D telepresence applications, enabling high-quality 3D depth data storage and transmission.

Virtual Tour Using Telepresence Robot and MQTT Protocol

We often face obstacles to visit certain places, facilities, and institutions. This was especially noticeable during the COVID-19 pandemic. In this period telepresence applications were of exceptional importance. They supported communications and provided social interactions, enriching our lives during isolation. In this paper, we propose a solution for telepresence using Padbot U1 robot and the MQTT protocol. The idea is to use this solution for virtual tours in museums, art galleries, historical sites, etc. We have developed two different applications. One is a client application used by the visitor, for controlling the robot remotely. The other application is for the mobile device (tablet) attached to the robot, which is used to enable telepresence and to forward control commands to the robot. By scanning the QR codes placed on the objects, visitors can also get specific information about each of objects. Communication between applications (sending commands and receiving information for objects) is realized with the MQTT protocol. For the developed solution, we also conducted experiments to evaluate some basic criteria for VTs.

Gestures vs. Emojis: Comparing Non-Verbal Reaction Visualizations for Immersive Collaboration.

Collaborative virtual environments afford new capabilities in telepresence applications, allowing participants to co-inhabit an environment to interact while being embodied via avatars. However, shared content within these environments often takes away the attention of collaborators from observing the non-verbal cues conveyed by their peers, resulting in less effective communication. Exaggerated gestures, abstract visuals, as well as a combination of the two, have the potential to improve the effectiveness of communication within these environments in comparison to familiar, natural non-verbal visualizations. We designed and conducted a user study where we evaluated the impact of these different non-verbal visualizations on users' identification time, understanding, and perception. We found that exaggerated gestures generally perform better than non-exaggerated gestures, abstract visuals are an effective means to convey intentional reactions, and the combination of gestures with abstract visuals provides some benefits compared to their standalone counterparts.

Online Avatar Motion Adaptation to Morphologically‐similar Spaces

AbstractIn avatar‐mediated telepresence systems, a similar environment is assumed for involved spaces, so that the avatar in a remote space can imitate the user's motion with proper semantic intention performed in a local space. For example, touching on the desk by the user should be reproduced by the avatar in the remote space to correctly convey the intended meaning. It is unlikely, however, that the two involved physical spaces are exactly the same in terms of the size of the room or the locations of the placed objects. Therefore, a naive mapping of the user's joint motion to the avatar will not create the semantically correct motion of the avatar in relation to the remote environment. Existing studies have addressed the problem of retargeting human motions to an avatar for telepresence applications. Few studies, however, have focused on retargeting continuous full‐body motions such as locomotion and object interaction motions in a unified manner. In this paper, we propose a novel motion adaptation method that allows to generate the full‐body motions of a human‐like avatar on‐the‐fly in the remote space. The proposed method handles locomotion and object interaction motions as well as smooth transitions between them according to given user actions under the condition of a bijective environment mapping between morphologically‐similar spaces. Our experiments show the effectiveness of the proposed method in generating plausible and semantically correct full‐body motions of an avatar in room‐scale space.

Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

Novel view synthesis has attracted tremendous research attention recently for its applications in virtual reality and immersive telepresence. Rendering a locally immersive light field (LF) based on arbitrary large baseline RGB references is a challenging problem that lacks efficient solutions with existing novel view synthesis techniques. In this work, we aim at truthfully rendering local immersive novel views/LF images based on large baseline LF captures and a single RGB image in the target view. To fully explore the precious information from source LF captures, we propose a novel occlusion-aware source sampler (OSS) module which efficiently transfers the pixels of source views to the target view’s frustum in an occlusion-aware manner. An attention-based deep visual fusion module is proposed to fuse the revealed occluded background content with a preliminary LF into a final refined LF. The proposed source sampling and fusion mechanism not only helps to provide information for occluded regions from varying observation angles, but also proves to be able to effectively enhance the visual rendering quality. Experimental results show that our proposed method is able to render high-quality LF images/novel views with sparse RGB references and outperforms state-of-the-art LF rendering and novel view synthesis methods.

Towards Wearable Augmented Reality in Healthcare: A Comparative Survey and Analysis of Head-Mounted Displays.

Head-mounted displays (HMDs) have the potential to greatly impact the surgical field by maintaining sterile conditions in healthcare environments. Google Glass (GG) and Microsoft HoloLens (MH) are examples of optical HMDs. In this comparative survey related to wearable augmented reality (AR) technology in the medical field, we examine the current developments in wearable AR technology, as well as the medical aspects, with a specific emphasis on smart glasses and HoloLens. The authors searched recent articles (between 2017 and 2022) in the PubMed, Web of Science, Scopus, and ScienceDirect databases and a total of 37 relevant studies were considered for this analysis. The selected studies were divided into two main groups; 15 of the studies (around 41%) focused on smart glasses (e.g., Google Glass) and 22 (59%) focused on Microsoft HoloLens. Google Glass was used in various surgical specialities and preoperative settings, namely dermatology visits and nursing skill training. Moreover, Microsoft HoloLens was used in telepresence applications and holographic navigation of shoulder and gait impairment rehabilitation, among others. However, some limitations were associated with their use, such as low battery life, limited memory size, and possible ocular pain. Promising results were obtained by different studies regarding the feasibility, usability, and acceptability of using both Google Glass and Microsoft HoloLens in patient-centric settings as well as medical education and training. Further work and development of rigorous research designs are required to evaluate the efficacy and cost-effectiveness of wearable AR devices in the future.

Real-Time 3D Reconstruction Method for Holographic Telepresence

This paper introduces a real-time 3D reconstruction of a human captured using a depth sensor and has integrated it with a holographic telepresence application. Holographic projection is widely recognized as one of the most promising 3D display technologies, and it is expected to become more widely available in the near future. This technology may also be deployed in various ways, including holographic prisms and Z-Hologram, which this research has used to demonstrate the initial results by displaying the reconstructed 3D representation of the user. The realization of a stable and inexpensive 3D data acquisition system is a problem that has yet to be solved. When we involve multiple sensors we need to compress and optimize the data so that it can be sent to a server for a telepresence. Therefore the paper presents the processes in real-time 3D reconstruction, which consists of data acquisition, background removal, point cloud extraction, and a surface generation which applies a marching cube algorithm to finally form an isosurface from the set of points in the point cloud which later texture mapping is applied on the isosurface generated. The compression results has been presented in this paper, and the results of the integration process after sending the data over the network also have been discussed.

Mixture of volumetric primitives for efficient neural rendering

Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a deconvolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.

Mixture of volumetric primitives for efficient neural rendering

Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a convolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.

The Plausibility Paradox for Resized Users in Virtual Environments

This paper identifies and confirms a perceptual phenomenon: when users interact with simulated objects in a virtual environment where the users’ scale deviates greatly from normal, there is a mismatch between the object physics they consider realistic and the object physics that would be correct at that scale. We report the findings of two studies investigating the relationship between perceived realism and a physically accurate approximation of reality in a virtual reality experience in which the user has been scaled by a factor of ten. Study 1 investigated perception of physics when scaled-down by a factor of ten, whereas Study 2 focused on enlargement by a similar amount. Studies were carried out as within-subjects experiments in which a total of 84 subjects performed simple interaction tasks with objects under two different physics simulation conditions. In thetrue physicscondition, the objects, when dropped and thrown, behaved accurately according to the physics that would be correct at that either reduced or enlarged scale in the real world. In themovie physicscondition, the objects behaved in a similar manner as they would if no scaling of the user had occurred. We found that a significant majority of the users considered themovie physicscondition to be the more realistic one. However, at enlarged scale, many users consideredtrue physicsto match their expectations even if they ultimately believedmovie physicsto be the realistic condition. We argue that our findings have implications for many virtual reality and telepresence applications involving operation with simulated or physical objects in abnormal and especially small scales.

Serverless streaming for emerging media: towards 5G network-driven cost optimization

Immersive 3D media is an emerging type of media that captures, encodes and reconstructs the 3D appearance of people and objects, with applications in tele-presence, teleconference, entertainment, gaming and other fields. In this paper, we discuss a novel concept of live 3D immersive media streaming in a serverless setting. In particular, we present a novel network-centric adaptive streaming framework which deviates from a traditional client-based adaptive streaming used in 2D video. In our framework, the decisions for the production of the transcoding profiles are taken in a centralized manner, by considering consumer metrics vs provisioning costs and inferring an expected consumer quality of experience and behaviour based on them. In addition, we demonstrate that a naive application of the serverless paradigm might be sub optimal under some common immersive 3D media scenarios.

Dynamic Point Cloud Denoising via Manifold-to-Manifold Distance.

3D dynamic point clouds provide a natural discrete representation of real-world objects or scenes in motion, with a wide range of applications in immersive telepresence, autonomous driving, surveillance, etc. Nevertheless, dynamic point clouds are often perturbed by noise due to hardware, software or other causes. While a plethora of methods have been proposed for static point cloud denoising, few efforts are made for the denoising of dynamic point clouds, which is quite challenging due to the irregular sampling patterns both spatially and temporally. In this paper, we represent dynamic point clouds naturally on spatial-temporal graphs, and exploit the temporal consistency with respect to the underlying surface (manifold). In particular, we define a manifold-to-manifold distance and its discrete counterpart on graphs to measure the variation-based intrinsic distance between surface patches in the temporal domain, provided that graph operators are discrete counterparts of functionals on Riemannian manifolds. Then, we construct the spatial-temporal graph connectivity between corresponding surface patches based on the temporal distance and between points in adjacent patches in the spatial domain. Leveraging the initial graph representation, we formulate dynamic point cloud denoising as the joint optimization of the desired point cloud and underlying graph representation, regularized by both spatial smoothness and temporal consistency. We reformulate the optimization and present an efficient algorithm. Experimental results show that the proposed method significantly outperforms independent denoising of each frame from state-of-the-art static point cloud denoising approaches, on both Gaussian noise and simulated LiDAR noise.

HeadJoystick: Improving Flying in VR Using a Novel Leaning-Based Interface.

Flying in virtual reality (VR) using standard handheld controllers can be cumbersome and contribute to unwanted side effects such as motion sickness and disorientation. This article investigates a novel hands-free flying interface-HeadJoystick, where the user moves their head similar to a joystick handle toward the target direction to control virtual translation velocity. The user sits on a regular office swivel chair and rotates it physically to control virtual rotation using 1:1 mapping. We evaluated short-term (Study 1) and extended usage effects through repeated usage (Study 2) of the HeadJoystick versus handheld interfaces in two within-subject studies, where participants flew through a sequence of increasingly difficult tunnels in the sky. Using the HeadJoystick instead of handheld interfaces improved both user experience and performance, in terms of accuracy, precision, ease of learning, ease of use, usability, long-term use, presence, immersion, sensation of self-motion, workload, and enjoyment in both studies. These findings demonstrate the benefits of using leaning-based interfaces for VR flying and potentially similar telepresence applications such as remote flight with quadcopter drones. From a theoretical perspective, we also show how leaning-based motion cueing interacts with full physical rotation to improve user experience and performance compared to the gamepad.

Addressing the Ethics of Telepresence Applications Through End-User Engagement.

Portacolone et al.'s Ethics Review highlights the ethical challenges associated with the implementation of telepresence devices and applications in the context of aging and dementia. In this response, we review ethical considerations as they relate to specific modalities of telepresence, with an emphasis on the continuum of potential interaction agents, from known individuals to fully automated and intelligent interlocutors. We further discuss areas in need of empirical evidence to inform regulatory efforts in telepresence. We close with a call for meaningful end-user engagement at all stages of technology development.

Immersive Mobile Telepresence Systems: A Systematic Literature Review

Telepresence can be defined as a system that provides remote collaboration between people in different locations, creating the feeling that both users share in fact the environment. The advances in communication, media and internet, has made possible the popularization of these systems. Smartphones have become increasingly powerful in processing, less expensive and more widespread. A single device combines various sensors, one or more cameras and internet connection, thus a potential hardware for telepresence applications. The main objective of this paper is to present a Systematic Literature Review to identify the main characteristics of immersive telepresence systems designed for mobile environment and to analyze research opportunities that can be further exploited or optimized. This research revealed that the development of immersive telepresence systems for mobile devices has increased in recent years, but is not yet widespread.

A Review of Telepresence, Virtual Reality, and Augmented Reality Applied to Clinical Care

This scoping review article explores the application of telepresence (TPr), virtual reality (VR), and augmented reality (AR) to clinical care. A literature search of key words was conducted from January 1990 through May 2019 of the following databases: PubMed/Medline, American Psychological Association PsycNET, Pubmed/Medline, Cochrane, Embase, PsycINFO, Web of Science, Scopus, OTSeeker, ABI/INFORM, computer-mediated communication (CMC), technology-mediated communications, Arts & Humanities Citation Index, Project Muse, ProQuest Research Library Plus, Sociological abstracts, Computers and Applied Sciences Complete and IT Source. It focused on concept areas: (1) TPr related to technologies; (2) virtual, augmented, reality, environment; (3) technology or computer-mediated communication; (4) clinical therapeutic relationship (boundaries, care, communication, connect, engagement, empathy, intimacy, trust); (5) telebehavioral health; (6) psychotherapy via technology; and (7) medicine/health care. Inclusion criteria were concept area 1 in combination with 2–7 and 2 or 3 in combination with any of 4–7. From a total of 5214 potential references, the authors found 512 eligible for full-text review and found 85 papers directly relevant to the concepts. From papers’ references and a review of books and popular literature about TPr, virtual reality (VR), and augmented reality (AR), 13 other sources of information were found. The historical evolution of TPr, VR, and AR shows that definitions, foci of studies (e.g., social neuroscience to business), and applications vary; assessments of TPr also vary widely. Studies discuss VR, AR, and TPr in medicine (e.g., rehabilitation, robotics), experimental psychology (laboratory, field, mixed), and behavioral health. Virtual environment (VE) designs aid the study of interpersonal communication and behavior, using standardized social interaction partners, virtual standardized patients, and/or virtual humans—all contingent on the participants’ experience of presence and the ability to engage. Additional research is needed to standardize experimental and clinical interventions, while maintaining ecological validity. Technology can significantly improve quality of care, access to new treatments and training, if ethical and reimbursement issues are better explored.

Image stitching by creating a virtual depth

A method for image stitching is presented. The approach focuses on images with parallax (depth variation) to create panoramic views with high fidelity. The approach creates the stitching seam at a virtual depth to convert hard stitching problems to simple ones. The virtual depth is created by applying local distortions to the input images at the stitching seam so that the contents visually appear to be located at the same depth. The presented approach targets a wide variety of applications that require generating high (or super) resolution, wide-view images. These applications include tele-presence (or tele-reality) applications such as shopping, touring, conferencing, planning or architecting, learning, inspection, and surveillance. Our results show that the proposed approach provides promising results compared to commercial products that rely on stitching solutions.

Real-time 3D volumetric human body reconstruction from a single view RGB-D capture device

Recently, volumetric video based communications have gained a lot of attention, especially due to the emergence of devices that can capture scenes with 3D spatial information and display mixed reality environments. Nevertheless, capturing the world in 3D is not an easy task, with capture systems being usually composed by arrays of image sensors, which sometimes are paired with depth sensors. Unfortunately, these arrays are not easy to assembly and calibrate by non-specialists, making their use in volumetric video applications a challenge. Additionally, the cost of these systems is still high, which limits their popularity in mainstream communication applications. This work proposes a system that provides a way to reconstruct the head of a human speaker from single view frames captured using a single RGB-D camera (e.g. Microsoft?s Kinect 2 device). The proposed system generates volumetric video frames with a minimum number of occluded and missing areas. To achieve a good quality, the system prioritizes the data corresponding to the participants? face, therefore preserving important information from speakers facial expressions. Our ultimate goal is to design an inexpensive system that can be used in volumetric video telepresence applications and even on volumetric video talk-shows broadcasting applications.

Using telepresence for social connection: views of older people with dementia, families, and health professionals from a mixed methods pilot study

Objectives: To explore the acceptability of telepresence robots in dementia care from the perspectives of people with dementia, family carers, and health professionals/trainees, and investigate the utility of a social presence assessment tool, the Modified-Temple Presence Inventory (Modified-TPI), for people with dementia.Method: A mixed-methods pilot study conducted in a social robotics laboratory. Three participant groups (n = 22) – dyads of people with dementia and their carers (n = 5 respectively), and health professionals/trainees (n = 12) – participated in individual one-off intervention sessions with the telepresence robot, Giraff, with follow-up interviews. Sessions covered how to use Giraff, followed by interactive practice in making a video-call involving conversation and manoeuvring of Giraff. Participants with dementia experienced receiving a call made by their carer; healthcare professionals/trainees experienced making and receiving a call. Outcomes of interest were sense of presence [Modified-TPI], affective response (International Positive and Negative Affect Schedule [I-PANAS-S]; Observable Displays of Affect Scale [ODAS]), and attitudes and reactions to Giraff (semi-structured interviews).Results: Participants reported a sense of authenticity and social connection through the experience. They indicated positive social presence through Giraff, and significantly higher positive (mean score 18.77; ±4.00) than negative affect (mean score 8.05; ±1.76) on the I-PANAS-SF, and on the facial display subscale of the ODAS (positive – mean score 15.50; ±3.51 versus negative – mean score 4.00; ±0.00).Conclusion: Telepresence has potential use in situations where people with dementia require social connection. Studies with larger sample sizes, varied characteristics, and cost-effectiveness analysis are needed to inform the application of telepresence in healthcare practice.