Multiple reference views for hole reduction in DIBR view synthesis

Abstract

Depth-image-based rendering (DIBR) technique is widely used to synthesize virtual views in many applications including multiview and free viewpoint video systems. An inherent problem of DIBR oriented view synthesis is that holes may appear in the synthesized view. Hole filling techniques generally extrapolates/interpolates the hole regions with the neighboring information based on an assumption that the texture pattern in the hole regions is similar to that of the neighboring background information. However, in some cases especially with complex texture, the assumption may not hold. Moreover, hole filling techniques can only provide an estimation for the hole regions which may not be true. In this paper, based on a thorough examination of the hole generation mechanism in view interpolation with multiple reference views, the problem of emerging holes in a target virtual view can be greatly alleviated by making good use of other neighboring complementary views in addition to its two (commonly used) most neighboring primary views. Furthermore, a scheme of selective warping of the complementary views is developed by efficiently locating a small number of useful pixels in the complementary views for hole reduction. Experimental results show that the hole size based on two primary reference views may be reduced by about 90 percent with the help of complementary reference views, while the developed selective warping scheme can save about 69 percent of rendering time compared with fully warping two complementary reference views in the rendering process.