Temporal Frame Interpolation Based on Multiframe Feature Trajectory

Abstract

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and to preserve the spatial correlation of MVs. Furthermore, the proposed method can detect occluded regions, where a continuously tracked feature does not have a corresponding point in the current frame. If an occluded region is detected, the intermediate frame is generated from either the previous frame or the current frame. In the non-occluded region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results showed that the proposed temporal frame interpolation (TFI) method can improve the visual quality compared with conventional TFI methods, both objectively and subjectively.