Video frame rate up conversion under inconsistent camera

Abstract

In this paper, we address the problem of video frame rate up conversion (FRC) in compressed domain. FRC is often recognized as video temporal interpolation. The problem is very challenging when targeted for a video sequence with an inconsistent camera and object motion. A novel compressed domain motion compensation scheme is presented and applied in this paper. The proposed algorithm uses MPEG-2 compressed motion vectors to undergo a cumulative spatiotemporal interpolation over a temporal sliding window of frames. An iterative rejection scheme based on the affine motion model is exploited to detect the global camera motion. Subsequently, the foreground object separation is performed by examining the temporal consistency of the output of iterative rejections. This consistency check process helps to conglomerate the resulting foreground macroblocks and weeds out the unqualified blocks, thus further refines the crude segmentation results. Finally, different strategies for compensating the camera motion and the object motion are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block based frame interpolation approach.