Tangible video teleconference system using real-time image-based relighting

Abstract

This paper deals with a real-time image based relighting system for tangible video teleconference. The proposed image based relighting system renders the extracted human object using the virtual environmental images. The proposed system can homogenize virtually the lighting environments of remote users on the video teleconference, or render the humans like they are in the virtual places. To realize the video teleconference, the paper obtains the reflectance models of users in real-time using the controlled lighting system. The reflectance models of human surface are obtained by a pair of lighting sources which are turned on and off alternatively. The images with different illuminations are exploited to model the reflectance on the object surfaces. The environment map for rendering is acquired by the mirror ball images or virtual background ones. And the relighting algorithm is implemented by GPU based programming for real-time operation. According to the simulations, the proposed system estimates the reflectance models of human faces, and relights the human faces in real-time. The objects are rendered by the reflectance models and virtual environment map. The proposed system enables the users to immerse video teleconference just as they are in the virtual environments.