Interactive Telepresence Research Articles

Integrated digital twin of networked immersive rooms for distributed interactive telepresence in shared virtual worlds

We present an integrated digital twin for room-oriented immersive systems (ROIS) that facilitates spatially distributed multi-user remote telepresence. This work, built upon prior development of game-based virtual collocation systems in immersive rooms, further addresses synchronous communication and spatially distributed interactivity between remote participants through a distributed audio spatialization system. This digital twin allows its users in physically disjoint locations to navigate through the virtual worlds with virtual sound source information transmitted through remote procedure calls (RPC). Precisioned user location tracking data is broadcast directly through in-game sessions. For each virtual sound source, generated procedurally or through non-invasive microphone inputs, real-time reverberation is synthesized and encoded based on proximity-modulatedperceptual parameters. These virtual sound sources are rendered in each ROIS facility through Open Sound Control (OSC). This approach offloads the otherwise expensive computational resources for physically based acoustic rendering techniques unsuited for networked applications. We demonstrate this by evaluating the system using latency metrics informed by round-trip-time measures. Taken as a whole, the integrated digital twin in this work also considers broader scalability so that researchers and content creators can deploy it for any immersive rooms alike in the future. [Work supported by NSF IIS-1909229 & CNS-1229391.]

Efficient neural implicit representation for 3D human reconstruction

High-fidelity digital human representations are increasingly in demand in the digital world, particularly for interactive telepresence, AR/VR, 3D graphics, and the rapidly evolving metaverse. Even though they work well in small spaces, conventional methods for reconstructing 3D human motion frequently require the use of expensive hardware and have high processing costs. This study presents HumanAvatar, an innovative approach that efficiently reconstructs precise human avatars from monocular video sources. At the core of our methodology, we integrate the pre-trained HuMoR, a model celebrated for its proficiency in human motion estimation. This is adeptly fused with the cutting-edge neural radiance field technology, Instant-NGP, and the state-of-the-art articulated model, Fast-SNARF, to enhance the reconstruction fidelity and speed. By combining these two technologies, a system is created that can render quickly and effectively while also providing estimation of human pose parameters that are unmatched in accuracy. We have enhanced our system with an advanced posture-sensitive space reduction technique, which optimally balances rendering quality with computational efficiency. In our detailed experimental analysis using both artificial and real-world monocular videos, we establish the advanced performance of our approach. HumanAvatar consistently equals or surpasses contemporary leading-edge reconstruction techniques in quality. Furthermore, it achieves these complex reconstructions in minutes, a fraction of the time typically required by existing methods. Our models achieve a training speed that is 110× faster than that of State-of-The-Art (SoTA) NeRF-based models. Our technique performs noticeably better than SoTA dynamic human NeRF methods if given an identical runtime limit. HumanAvatar can provide effective visuals after only 30 s of training. Please visit https://github.com/HZXu-526/Human-Avatar for further demo results and code.

Tele-Presence Robot for Medical Assistance in Hospitals

The visual interactive Tele-presence robot is used to take care of COVID-19 patients remotely in order to avoid the pandemic. This aims to monitor the body temperature, to deliver medicines to the patients as per doctor’s instruction and to disinfect floors and bed in isolation wards without manual intervention. This prototype consists of various parts including raspberry-pi processor, ultrasonic sensor, webcam and tablet. The key idea of this prototype is to perform visual communication through video conferencing with the help tablet computer. The movement of the robot can be controlled by sending commands through computer. As per given command, the robot will move freely in barrier-free environment. However, in the event of a barrier environment, the robot is properly guided by an ultrasonic sensor mounted on the robot. The Raspberry Pi processor is used to keep track of the robotics.

Exploring the Feasibility of Wearable Technologies to Provide Interactive Telepresence Sub-Specialist Support to Remote Clinicians Treating Patients with Traumatic Injuries

Introduction:Some patients presenting to rural or regional hospitals may be deteriorating so rapidly that emergency procedures might be necessary before transfer to specialist facilities. Such interventions might include placement of an ICC, establishing a surgical airway, evacuation of an EDH, laparotomy, or intra-abdominal packing. The treating clinician may have had little or no experience in the procedure. Interactive telepresence technology offers further point of care support to the treating clinicians through the virtual presence of a specialist from a major trauma center.Aim:To explore the feasibility of wearable interactive telepresence technology that can provide sub-specialist support to remote clinicians treating patients with traumatic injuries.Methods:Thirty-seven wearable near-field display devices and annotation software applications were tested against a set of pre-specified technical and user experience requirements. A shortlist of three devices and two software applications underwent usability evaluations with a convenience sample of 24 junior clinicians and sub-specialists. The junior clinicians trialed the wearable devices and the sub-specialists trialed the annotation applications in three simulated trauma scenarios. Measures included participants’ ratings of acceptance and workload, technical issues encountered (e.g. frequency of call drop-outs), and anecdotal comments.Results:Participants’ subjective ratings of the solutions and anecdotal feedback were positive. However, there was no clear solution that satisfied the functionality and ease-of-use requirements for all participants. For example, the solutions that were rated more favorably by the junior clinicians were rated less favorably by the sub-specialists, and vice versa.Discussion:This work provided preliminary evidence of the feasibility and usefulness of interactive telepresence technology in healthcare. A second phase of usability testing is currently underway to explore additional device and software combinations, including those with augmented reality functionality. Future phases of the project will evaluate the solutions under higher-fidelity conditions followed by in-situ trials across selected regional centers.

Classification and Analysis of 3D Teleimmersive Activities

To provide users with a high-quality experience, interactive telepresence system platforms must accommodate multiple performance profiles for diverse, shared cyberphysical activities.

Depth map compression for real-time view-based rendering

Realistic and interactive telepresence has been a hot research topic in recent years. Enabling telepresence using depth-based new view rendering requires the compression and transmission of video as well as dynamic depth maps from multiple cameras. The telepresence application places additional requirements on the compressed representation of depth maps, such as preservation of depth discontinuities, low complexity decoding, and amenability to real-time rendering using graphics cards. We propose an adaptation of an existing triangular mesh generation method for depth representation that can be encoded efficiently. The mesh geometry is encoded using a binary tree structure where single bit enabled flags that mark the split of triangles and the depth values at the tree nodes are differentially coded. By matching the tree traversal to the mesh rendering order, both depth map decoding and triangle strip generation for efficient rendering are achieved simultaneously. The proposed scheme also naturally lends itself to coding segmented foreground layers and providing error resilience. At similar compression ratio, new view generation using the proposed method provided similar quality as depth compression using JPEG2000. However, the new mesh based depth map representation and compression method showed a significant improvement in rendering speed when compared to using separate compression and rendering processes.