Learning-Based Stereoscopic View Synthesis with Cascaded Deep Neural Networks

Abstract

Depth image-based rendering (DIBR) is an important technique in the 2D to 3D conversion process, which renders virtual views with a texture image and the associated depth map. However, certain problems, such as disocclusion, still exist in current DIBR systems. In this study, a new learning-based framework that models conventional DIBR synthesis pipelines is proposed to solve these problems. The proposed model adopts a coarse-to-fine approach to realize virtual view prediction and disocclusion region refinement sequentially in a unified deep learning framework that includes two cascaded joint filter block-based convolutional neural networks (CNNs) and one residual learning-based generative adversarial network (GAN). An edge-guided global looping optimization strategy is adopted to progressively reconstruct the scene structures on the novel view, and a novel directional discounted reconstruction loss is proposed for better training. In this way, our framework performs well in terms of virtual view quality and is more suitable for 2D to 3D conversion applications. The experimental results demonstrate that the proposed method can generate visually satisfactory results.