Optical Flow-Guided Multi-Scale Dense Network for Frame Interpolation

Abstract

Video frame interpolation is a traditional computer vision task, which aims to generate intermediate frames between two given consecutive frames. Many algorithms attempt to solve this task relying on optical flow to compute dense pixel correspondence. According to the estimated flow, the input images are warped to the location of the interpolated frame, and then blended together to generate synthesis frame. However, due to the difficulty of flow estimation, this method always leads to blurry region and visually unpleasant results. To overcome the limitation of inaccurate flow estimation, we perform an end-to-end neural network to improve interpolation results after warping, which explicitly uses optical flow but not completely depends on it. Moreover, we design a multi-scale dense network for frame interpolation (FIMSDN), which not only makes full use of the multi-scale information for large motion frame interpolation, but also strengthens feature propagation. Specifically, a pre-trained optical flow net is firstly utilized to produce the bidirectional flow between two input frames. The input images are warped to the middle frame by the estimated flow and then fed with the original images into the FIMSDN to directly estimate the in-between frame. Experimental results show the improvement in terms of both objective and subjective quality by comparing with other recent optical flow and convolutional neural network (CNN) based methods.