Simultaneous color-depth super-resolution with conditional generative adversarial networks

Abstract

In this paper, color-depth conditional generative adversarial networks (CDcGAN) are proposed to resolve the problems of simultaneous color image super-resolution and depth image super-resolution in 3D videos. Firstly, a generative network is presented to leverage the mutual information of the low-resolution color image and low-resolution depth image so that they can enhance each other considering their geometric structural similarity in the same scene. Secondly, three auxiliary losses of data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to ensure that the generated images are close to the real ones in addition to the adversarial loss. Finally, we study the CDcGAN and its variants. Experimental results show that the proposed approach can produce the high-quality color image and depth image from a pair of low-quality images, and it is superior to several other leading methods. Additionally, it has also been used to resolve the problems of concurrent image smoothing and edge detection, as well as the problem of HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method.