Self-Supervised Depth Enhancement

Abstract

Most of the existing learning-based depth enhancement techniques train in a supervised fashion. These techniques extensively rely either on synthetic datasets for noise and artefact free depth maps or, on pre-processing of the real-world depth maps to mimic artefact-free depth maps. The results from these techniques although promising might not optimally generalize to the vagaries of real-world settings. In this paper, we propose a technique to enhance depth maps in a self-supervised way, which enables us to work directly on the real-world depth maps captured by commercial RGB-D cameras which inherently contains artefacts. To achieve self-supervision, captured depth maps are initially degraded by adding more hole-artefacts based on edge-information obtained from the corresponding RGB image, the training is then done by allowing the model to learn to estimate depth in the regions of holes by a masking process. Through generalization across a large training set, we are able to predict accurate depth values in the regions of hole-artefacts without even observing a single artefact-free depth map during training. The proposed method is able to circumvent the dependence on synthetic datasets and/or complicated setups to capture accurate depth maps by working only on real-world, incomplete data in a self-supervised manner.