Segmentation-based view synthesis for multi-view video plus depth

Abstract

In this paper, we present a novel view synthesis algorithm for three-dimensional video. The proposed algorithm is based on segmentation using multi-level thresholding method. Recently, numerous techniques have been suggested which use a 2-D color image and the per-pixel depth map of the scene to create virtual views of the scene from any viewing position. However, inaccuracy in the depth maps cause annoying visual artifacts in depth-based view synthesis. In the proposed method, the depth maps are first preprocessed to avoid the errors caused by wrong depth values. Then, the color images are segmented according to the depth values and the regions belonging to different segments are warped independently. To further enhance the quality of the synthesized views, a multi-level thresholding based ghost removal algorithm and a novel hole filling algorithm have been proposed. Experimental results show that the proposed methods achieve an average PSNR gain of 0.98 dB for the multi-view test sequences and also improve the subjective quality of the synthesized views.