Abstract
We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.
Highlights
Improvements in telecommunications have steadily increased both the fidelity and availability of synchronous communication over long-distance networks [Sterling and Shiers 2000]
We find that an acceptable level of visual quality is obtained by setting the codec quantization parameter (QP) to 14 for depth data and to 22 for color
We conducted a controlled, within-subjects experiment in which each participant had a conversation in both our system and traditional videoconferencing, so we could statistically test for communication variables, as well as move beyond self-report data to include behavioral measures like body language and memory recall
Summary
Improvements in telecommunications have steadily increased both the fidelity and availability of synchronous communication over long-distance networks [Sterling and Shiers 2000]. Video-based systems like Skype, FaceTime, Zoom, Meet, and Teams are a recent step forward in bringing people closer together who are far apart. At the far end of this spectrum is telepresence, i.e., enabling remote participants to feel copresent, as if they are occupying a shared. Telepresence presents tremendous opportunities to bring together the world’s increasingly distributed organizations and social groups. (1) Capture and render a 3D audiovisual likeness of a remote person, so realistic that one forgets it is not real. (3) Achieve copresence — the feeling that two people are together — including proximity, eye contact, and interaction. We demonstrate a telepresence system representing a significant milestone along these different dimensions. User studies demonstrate an improved experience over traditional 2D videoconferencing
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
