Discrete and continuous optimizations for depth image super-resolution

Abstract

Recently a Time-of-Flight 2D/3D image sensor has been developed, which is able to capture a perfectly aligned pair of a color and a depth image. To increase the sensitivity to infrared light, the sensor electrically combines multiple adjacent pixels into a depth pixel at the expense of depth image resolution. To restore the resolution we propose a depth image super-resolution method that uses a high-resolution color image aligned with an input depth image. In the first part of our method, the input depth image is interpolated into the scale of the color image, and our discrete optimization converts the interpolated depth image into a high-resolution disparity image, whose discontinuities precisely coincide with object boundaries. Subsequently, a discontinuity-preserving filter is applied to the interpolated depth image, where the discontinuities are cloned from the high-resolution disparity image. Meanwhile, our unique way of enforcing the depth reconstruction constraint gives a high-resolution depth image that is perfectly consistent with its original input depth image. We show the effectiveness of the proposed method both quantitatively and qualitatively, comparing the proposed method with two existing methods. The experimental results demonstrate that the proposed method gives sharp high-resolution depth images with less error than the two methods for scale factors of 2, 4, and 8.