Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network

Abstract

Depth image super-resolution (DISR) is an effective solution to improve the quality of depth images captured by real world low-cost cameras. In this paper, we propose a multi-scale symmetric network with the correlation-controlled color guidance block (CCGB) for DISR. The proposed network consists of two multi-scale sub-networks to respectively provide guidance and estimate depth. A symmetric unit (SU), which is a mini-encoder-decoder structure with residual learning, is designed and used as a basic network atom. The encoder part in SU aims to extract essential features, while the decoder part works to restore edge details. The way the SU processes information matches well with the textureless and sharp-edge characteristics of depth images. The two sub-networks present a high-level symmetric structure connected by dense guidance links in between. Based on the correlation analyses between the two sub-networks, each guidance link will transfer information trough a CCGB designed to implement channel-wise re-weighting mechanism. The accurate color guidance from CCGB helps avoiding artifacts introduced by non-co-occurrence of depth discontinuities and color edges. Experimental results demonstrate the superiority of the proposed method over several state-of-the-art DISR works.