Depth map propagation with the texture image guidance

Abstract

We propose a novel method to propagate the depth maps from key-frames to the non-key frames in a video shot. The depths of non-key frames are initialized by warping the depth maps of their corresponding key frames according to the motion estimation between them. Such initial depth maps are then refined with the guidance of their texture frames in a convex optimization process. The energy function in the optimization involves three kinds of constraints, (i) the similarity between the initial depths and the estimated ones, (ii) the spatial constraint to maintain the smoothness of homogeneous regions and region boundaries, and (iii) the temporal constraint to guarantee the temporal smoothness between adjacent frames. In the experiments, we evaluate our method on six video sequences with ground truth depth maps. The experiment results show that our method is robust to many complex scenes and obtain lower error rates of the depth propagation than the baseline method.