Surface normal data guided depth map restoration with edge-preserving smoothing regularization

Abstract

High-quality depth information is urgently required with their increasingly wide application in many real-world multimedia fields. However, due to the limitation of depth sensing technology, the captured depth map in practice usually owns low resolution and poor quality, which limits its practical application. As we all know, consistency between high-quality color images and low-quality depth maps achieves good effects in depth super-resolution. But the edge inconsistency also limits the recovery of depth map. Inspired by the geometric relationship between surface normal of a 3D scene and their distance from camera, we discover that there are more consistency between surface normal map and depth map in the edge areas. Meanwhile, surface normal map can provide more spatial geometric constraints for depth map reconstruction, for both of them are special images with spatial information, which we called 2.5D images. In this paper, we propose a unified framework, Normal Data Guided Depth Map Restoration with Edge-Preserving Smoothing Regularization (NDEPS) method, via joint spatial domain and gradient domain regularization, one characterizing the relationship between surface normal data and depth in the spatial domain and another edge-aware constraint in the gradient domain. The proposed NDEPS method formulates a constrained optimization problem that can be solved by an iterative conjugate gradient(CG) algorithm. Extensive quantitative and qualitative evaluations compared with state-of-the-art depth recovery methods show the effectiveness and superiority of our method.