A Framework for Constructing Real-time Immersive Environments for Training Physical Activities

Abstract

This paper describes a framework for constructing a three-dimensional immersive environment that can be used for training physical activities. The system is designed to capture three dimensional full-body human motion in real time and visualize the data either locally or remotely through three- dimensional display system so that the data can be viewed from arbitrary viewpoints. In the proposed system, an immersive environment is constructed through realistic reconstruction of a scene by applying a stereo algorithm on a set of images that is captured from multiple viewpoints. Specifically, twelve camera clusters that consist of four camera quadruples are used to capture the scene, where each camera cluster is processed by a pc independently and synchronously so that partial reconstructions from each viewpoint are merged to form a complete 3D description of the scene. This paper discusses in detail system architectures that enable synchronous operations across multiple computers while achieving parallel computations within each multi-processor system. A set of experiments is performed to learn tai-chi lessons in this environment where students are instructed to follow pre-recorded teacher’s movements while observing both their own motions and the teacher in real-time. The effect of learning in the virtual environment is discussed by comparing the performance of the trainee groups under different control environments.