A Multi-frame Video Interpolation Neural Network for Large Motion

Abstract

Video frame interpolation algorithms typically estimate optical flow to guide the synthesis of intermediate frame(s) between two consecutive input frames. However, the estimation of optical flow is easily affected by large motion. To tackle this problem, we combine multi-scale optical flow network PWC-Net and optimized network UNet++ to form our multi-frame interpolation neural network, which can be trained end-to-end. Specifically, we first use PWC-Net to estimate bidirectional optical flows between two input frames and linearly combinate the flows at each time step to obtain the approximate flows for generating the intermediate frames. Next, we use a modified UNet++ to refine the approximate flows and avoid the effects of occlusion. Finally, guided by the accurate flows, two input frames are warped and linearly fused to form each intermediate frame. Experiments show that our network outperforms representative state-of-the-art methods, especially in large motion scenarios.