Shaking video stabilization with content completion

Abstract

ABSTRACT A new stabilization algorithm to counterbalance the shaking motion in a video based on classical Kandade-Lucas-Tomasi (KLT) method is presented in this paper. Feature points are evaluated with law of large numbers and clustering algorithm to reduce the side effect of m oving foreground. Analysis on the change of motion direction is also carried out to detect the existence of shaking. For video clips with det ected shaking, an affine transformation is performed to warp the current frame to the reference one. In addition, the missing content of a frame during the stabilization is completed with optical flow analysis and mosaicking operation. Experiments on video clips demonstrate the effectiveness of the proposed algorithm. Keywords: video Stabilization, feature matching, content completion 1. INTRODUCTION The movement of camera can result in various inter-frame motions such as shaking in videos, which will definitely decrease the visual quality of the videos. For example, videos taken by hand-hold cameras or on a moving vehicle usually have such motion. With the popularization of digital camcorders, it becomes more and more important to develop an effective algorithm to stabilize the videos with shaking. The key issue of video stabilization is to acquire the accurate inter-frame motion parameters. That is to say, the accuracy of motion estimation has direct influence on video stabilization. However, the reliability of algorithm would greatly reduce if the features of foreground objects involve in motion estimation. In [1], Shi and Tomasi proposed a method to select feature points by using a measure of feature dissimilarity that quantified the change of appearance of features between the reference and the current frame. In [2], Hu et.al adopted scale invari ant feature transforma tion (SIFT) as the features to estimate the inter-frame transf orm for video stabilization. Although there have already been so many feature matching and tracking methods [1][2][3], the side effects of foreground objects, which will cause the error of global motion estimation, are still an open problem. On the other hand, unnecessary computing can be caused if the video clip has no shaking. Therefore, it is necessary to detect if there exists inter-frame motion in the current frame before doing video stabilization. In [4], Yan et al. proposed a shaking detection algorithm based on the moving direction analysis of objects, however, it can not detect shaking when intentional motion is large and shaking displacement is small. In [5], Anberger et al. put forward a method to remove translation and jitter from video sequence by local and global motion compensation. The method directly does video stabilization without judging while we think the shaking detection should be carried out at first. Another important issue in video stabilization is that the stabilized video frames may produce undefined areas because of motion compensation. Litvin et al. [6] used mosaicking to re construct undefined areas. Matsus hita et al. [7] presented a method called motion inpainting to fill the missing areas. Chen et al. [8] proposed a method which fills dynamic and static missing areas to keep the same resolution and quality as the original video considering users’ capturing intention. Although these methods can produce good results in most cases, they might fail when speedily moving foreground objects appear at the border of the video frame.