Trilateral constrained sparse representation for Kinect depth hole filling

Abstract

Due to measurement errors or interference noise, Kinect depth maps exhibit severe defects of holes and noise, which significantly affect their applicability to stereo visions. Filtering and inpainting techniques have been extensively applied to hole filling. However, they either fail to fill in large holes or introduce other artifacts near depth discontinuities, such as blurring, jagging, and ringing. The emerging reconstruction-based methods employ underlying regularized representation models to obtain relatively accurate combination coefficients, leading to improved depth recovery results. Sparse representation facilitates retaining the saliency features of natural images and is thus more favorite than other regression models in image restoration, e.g. ridge regression. However, its naive applicability to depth map recovery hardly affords satisfactory depth prediction. Motivated by locality learning and bilateral filtering, this paper advocates a trilateral constrained sparse representation for Kinect depth recovery, which considers the constraints of intensity similarity and spatial distance between reference patches and target one on sparsity penalty term, as well as position constraint of centroid pixel in the target patch on data-fidelity term. Learning from the accompanied color image, this method can produce optimal solution to hole-filling problem in terms of depth prediction accuracy. Various experimental results on real-world Kinect maps and public datasets show that the proposed method outperforms state-of-the-art methods in filling effects of both flat and discontinuous regions.