Fine-Grained Motion Estimation for Video Frame Interpolation

Abstract

Recent advances in video frame interpolation have shown that convolutional neural networks combined with optical flow are capable of producing a high-quality intermediate frame between two consecutive input frames in most scenes. However, existing methods have difficulties dealing with large and non-uniform motions that widely exist in real-world scenes because they often adopt the same strategy to deal with different motions, which easily results in unsatisfactory results. In this article, we propose a novel fine-grained motion estimation approach (FGME) for video frame interpolation. It mainly contains two strategies: multi-scale coarse-to-fine optimization and multiple motion features estimation. The first strategy is to gradually refine optical flows and weight maps, both of which are used to synthesize the target frame. The second strategy aims to provide fine-grained motion features by generating multiple optical flows and weight maps. To demonstrate its effectiveness, we propose a fully convolutional neural network with three refinement scales and four motion features. Surprisingly, this simple network produces state-of-the-art results on three standard benchmark datasets and real-world examples, with advantages in terms of effectiveness, simplicity, and network size over other existing approaches. Furthermore, we demonstrate that the FGME approach has good generality and can significantly improve the synthesis quality of other methods.