Bandelet-based video inpainting

Abstract

In this work, a framework for inpainting missing parts of a video sequence is presented. The first step in the algorithm consists of finding the geometrical flow of each frame in the video sequence by using the Bandelet transform. Consequently, each frame is segmented into a quadtree where each dyadic square regroups pixels sharing the same geometrical flow direction. Then, two cases are considered. The first case is concerned with the removal of non-stationary objects that occlude stationary background. For this, the missing portions of the video sequence are filled by searching for pairs of dyadic squares in the successive frames quadtrees that are valid neighbors to the dyadic squares at the boundary of the inpainting zone. The search for valid neighbors squares is accomplished by the minimization of a cost measure that studies the geometrical similarity of each pair of squares. Once neighbors squares are identified, the corresponding squares in the original frame images are used to fill-in the missing areas. The second case involves filling-in moving objects when they are partially occluded. For this, we propose a priority scheme to first inpaint the occluded moving objects and then fill-in the remaining area with stationary background using the method proposed for the first case. Optical flow is used here, which tells if an undamaged pixel is moving or is stationary. We evaluate the performance of our algorithm based on a set of video sequences with different types of occlusions.