A cluster based free viewpoint video system using region-tree based scene reconstruction

Abstract

Free viewpoint video (FVV) has been widely speculated as one of the next generation of visual media applications. By taking advantage of camera array based multiple imaging techniques, FVV enables free viewpoint navigation to invoke a sense of “being immersed” for the viewers. This thesis presents a cluster based FVV system which is designed as a specific application using a new proposed framework for general camera array applications. Our FVV system enables centralized workflow management and distributed computation to take advantage of the cluster’s computation power for fast FVV-oriented video processing. For its implementation, effort is mainly focused on the FVV workflow stages of multi-view video acquisition and dense depth based scene reconstruction. Specifically, a new automatic method is proposed for the efficient geometric, photometric and temporal calibrations of a camera array. With this novel integrated calibration method, the use of unsynchronized cameras becomes possible and the multi-view video acquisition is made easy, which greatly facilitate the practical use of a FVV (or camera array based) system. On the other hand, the dense depth based FVV scene reconstruction is addressed as an image discrete labeling problem using a novel coarse-to-fine region-tree based framework. As a general framework, its high ranking evaluations in standard binocular stereo matching and optical flow estimation benchmarking show its effectiveness and versatility. By further extending it for general position multi-view temporal stereo and integrating with inconsistency map/background based progressive optimization, spatial-temporal consistency is enforced in a new and unified way, which greatly helps the final FVV rendering quality. Extensive experimental results show that the new system with its accompanying algorithms can provide high quality rendering results.