Interframe Interpolation of Cinematic Sequences

Abstract

We present a new algorithm for interframe interpolation of cinematic sequences. We demonstrate its applicability to video data compression of pedestrian traffic and to data compression for video conferencing. In both of these applications it is assumed that the background is nearly stationary and that there are no interobject occlusions. The interpolation algorithm makes use of estimates of optical flow to compensate for the motion of objects between two frames. We describe three major problems associated with motion compensated cinematic interpolation: interframe occlusion, interframe zooming, and figure-ground ambiguity. Our algorithm suppresses artifacts caused by all three of these problems. Artifacts of interframe occlusion are suppressed by detecting mismatches between forward and backward optical flow. Artifacts of interframe zooming are suppressed by reversing the temporal order of the frames. Artifacts of figure-ground ambiguity are suppressed by exploiting the near-stationarity of the background. We tested our algorithm on a video sequence of pedestrians, subsampled by 3 to 1, and on a video sequence of a head-and-shoulders view of a salesman, subsampled by 4 to 1, and compared the resulting sequences to the sequences produced by two other algorithms: (a) the matching of forward and backward optical flow as described by Cafforio et al., and (b) linear interpolation. We evaluated the distortion quantitatively by computing the histograms of the relative absolute error. Our results are qualitatively and quantitatively superior to those produced by these other two algorithms.