Learning Modal and Spatial Features With Lightweight 3D Convolution for RGB Guided Depth Completion

Abstract

RGB guided depth completion aims to recover a complete depth map from a sparse set of depth measurements and one corresponding RGB image, which is efficient for 3D applications to generate high-quality depth maps. Most prevailing approaches feed the sparse depth data and RGB image collected by consumer devices into a 2D convolutional network performed only at the spatial level. We argue that there is a correlation in registered multi-modal data between different modalities, which is ignored in 2D convolutional operations, resulting in loss of accuracy. In order to acquire the extra modal information between different modalities, we adopt 3D convolution for the depth completion task. Meanwhile, to decrease the significantly increased parameter size 3D convolutions, we propose a simple and effective method to reduce this increase while retaining their modal information. We verified the effectiveness of our proposed method for modal and spatial features learning on NYUv2 and KITTI depth completion datasets. Our lightweight 3D convolution achieved approximate accuracy as the standard 3D convolution, but with the same parameter size of 2D convolution. Our proposed modal features and lightweight 3D convolution are helpful to inform the development of depth sensors for consumer devices.