Tele-immersive Systems Research Articles

Weighted Large Margin Nearest Center Distance-Based Human Depth Recovery With Limited Bandwidth Consumption.

This paper proposes a weighted large margin nearest center (WLMNC) distance-based human depth recovery method for tele-immersive video interaction systems with limited bandwidth consumption. In the remote stage, the proposed method highly compresses the depth data of the remote human into skeletal block structures by learning the WLMNC distance, which is equivalent to downsampling the human depth map at $64{\times}$ the sampling rate. In the local stage, the method first recovers a rough human depth map based on a WLMNC distance augmented clustering approach and then obtains a fine depth map based on a rough depth-guided autoregressive model to preserve the depth discontinuities and suppress texture copy artifacts. The proposed WLMNC distance is learned by the large margin clustering problem with a weighted hinge loss to balance the clustering accuracy and depth recovery accuracy and is verified to be able to preserve depth discontinuities between skeletal block structures with occlusion. A theoretical analysis is conducted to verify the effectiveness of using the weighted hinge loss. Furthermore, a novel data set containing various types of human postures with self-occlusion is built to benchmark the human depth recovery methods. The quantitative comparison with the state-of-the-art depth recovery methods on the introduced benchmark data set demonstrates the effectiveness of the proposed method for human depth recovery with such a high upsampling rate.

Tele-Immersion Concepts.

Tele-immersive systems development is always driven as well as restricted by the available immersive technology. Hence, existing such systems are described mainly from a technological point of view; their conceptual description is usually limited to the description of a scenario that is implementable with or circumvents the limitations of the chosen technology. This focus on technology makes it difficult to compare systems' concepts; moreover, it has led to different views on tele-immersion in different fields, such as remotely controlled robots, immersive video conferencing, and tele-collaboration. In this work, we give a general, structured principle to describe the conceptual part of any tele-immersion system. This principle naturally unifies the different views on tele-immersion. Our idea is based on the insight that, in order to be general, immersion must be described separately for each direction of communication. We characterize communication between locations using a graph; for each directed edge of this graph, we describe immersion as operations on volumes. Using this principle, we define a typology, which enables the comparison and enumeration of tele-immersion concepts. We apply this typology to survey the concepts of existing tele-immersion systems and thereby demonstrate how three well-known tele-immersive scenarios-Marvin Minsky's tele-operated robot, the Office of the Future, and the asymmetric Beaming scenario-integrate naturally. We show how the general principle can be utilized conveniently to grasp conceptual ideas in tele-immersion, such as direct interaction, locational presence, spatial consistency, symmetries, and self-inclusion.

Towards coordinated bandwidth adaptations for hundred-scale 3D tele-immersive systems

3D tele-immersion improves the state of collaboration among geographically distributed participants. Unlike the traditional 2D videos, a 3D tele-immersive system employs multiple 3D cameras based in each physical site to cover a much larger field of view, generating a very large amount of stream data. One of the major challenges is how to efficiently transmit these bulky 3D streaming data to bandwidth-constrained sites. In this paper, we study an adaptive Human Visual System (HVS) -compliant bandwidth management framework for efficient delivery of hundred-scale streams produced from distributed 3D tele-immersive sites to a receiver site with limited bandwidth budget. Our adaptation framework exploits the semantics link of HVS with multiple 3D streams in the 3D tele-immersive environment. We developed TELEVIS, a visual simulation tool to showcase a HVS-aware tele-immersive system for realistic cases. Our evaluation results show that the proposed adaptation can improve the total quality per unit of bandwidth used to deliver streams in 3D tele-immersive systems.

Tele-immersive environments for rehabilitation activities: an empirical study on proprioception

Those with proprioceptive impairments can use their other senses as the proprioceptive feedback. We hypothesize that tele-immersion systems can aid in this supplemented proprioception by providing novel visual perspectives of one's own body. In particular we are interested as to whether the real-time 3D reconstructions used in tele-immersive systems are better or worse than conventional 2D views (i.e. video) with regards to aiding a task requiring proprioception. The objective of this study is to investigate the stated hypothesis by quantifying and ranking the various visual and auditory cues available in a tele-immersive system as they are used during an assigned task. The paper briefly describes a portable immersive VR system, our methodology for quantifying task performance, and results from our experiments with wheelchair basketball athletes.

Human-centric control of video functions and underlying resources in 3d tele-immersive systems

3D tele-immersion (3DTI) has the potential of enabling virtual-reality-like interaction among remote people with real-time 3D video. However, today's 3DTI systems still suffer from various performance issues, limiting their broader deployment, due to the enormous demand on temporal (computing) and spatial (networking) resources. Past research focused on system-centric approaches for technical optimization, without taking human users into the loop. We argue that human factors (including user preferences, semantics, limitations, etc.) are an important and integral part of the cyber-physical 3DTI systems, and should not be neglected.