Layered approach for improving the quality of free-viewpoint depth-image-based rendering images

Abstract

In free-viewpoint rendering systems, one of the most challenging goals is the creation of virtual views based on available color texture (RGB) and depth data. Conventional depth-image-based rendering (DIBR) approaches have assumed that the virtual camera can only be displaced horizontally, thus leading to fairly simple disocclusion artifacts. However, in free-viewpoint DIBR, the virtual camera can be positioned in an arbitrary way and the respective disocclusion artifacts can exhibit complicated anisotropic appearances. Consequently, conventional approaches for compensating disocclusion holes usually fail in such arbitrary camera motion. We present a disocclusion compensation technique based on texture inpainting. We propose a layered representation of both the color and depth images in local foreground, background, and undefined segments (a trimap). This representation allows for employing an efficient alpha-matting approach for reconstructing the underlying opacity layer followed by a background compensation and layered rendering. The performance of the proposed method is evaluated with respect to the state-of-the-art through objective and subjective tests. The achieved results, especially for large camera displacements, outperform the state-of-the-art. Those results assess the effectiveness of the proposed method and highlight the need for new quality metrics able to address the impairments of this type of content.