Magnification-arbitrary depth super-resolution with multiscale consistency deformable alignment

Abstract

For many years, depth map super-resolution has been a problem worthy of studying in 3D reconstruction. Although many excellent algorithms have been developed to solve this problem, depth map super-resolution at any scale has not attracted much attention. In this paper, we propose a magnification-arbitrary depth super-resolution method based on Meta-SR framework. Different from the single spatial location guidance in Meta-SR, the multiscale consistency alignment module learns a hierarchical registered discontinuities and semantic feature guidance to aggregate more structural information of the depth map and its corresponding color image. Specially, the dense pyramid deformable convolution realizes the scale expansion and improves the precision of sub-pixel upsampling from coarse to fine. A collaborative geometrical-spatial attention based weight prediction module is proposed to measure the correlation between geometrical features, spatial location and the upsampling weights, and predict a set of weight filters for reconstruction. A large number of experiments show that our method achieves satisfactory subjective and objective results on both integral and fractional scales compared with existing methods.