Graph-based interpolation for DIBR-synthesized images with nonlocal means

Abstract

Given texture and depth maps of a single reference viewpoint, one can synthesize a novel viewpoint image via depth-image-based rendering (DIBR) by mapping texture pixels from reference to virtual view. When the virtual viewpoint is much closer to the 3D scene than the reference view (camera movement in the z-dimension), objects close to the camera will enlarge in size in the virtual viewpoint image. An object's enlargement during DIBR means that its pixel samples in the reference view will be scattered to a larger spatial area, resulting in expansion holes. Following our previous work, we investigate the problem of expansion hole completion. We first assume a previously proposed method based on depth histogram is used to identify missing or erroneously translated pixels as expansion holes. We then propose a new graph-based interpolation technique to fill in expansion holes. Unlike our previous work, nonlocal but similar pixel patch information are incorporated into a new graph construction before a graph-based interpolation procedure with sparsity prior is executed, resulting in enhanced performance. Experimental results show that our new procedure of expansion hole filling can outperform inpainting procedure employed in VSRS 3.5 by up to 4.02 dB.