Multiframe interpolation for video using phase features

Abstract

Traditional frame interpolation algorithms typically find dense correspondences to synthesize an in-between frame. Finding correspondences is often sensitive to occlusion, disocclusion, and changes in color or luminance. We present a phase-feature-aided multiframe interpolation network that aims to estimate multiple in-between frames in one pass and handle challenging scenarios such as extreme light changes and occlusion. We first model the relation between multiple in-between frames together to enhance the temporal consistency. Two candidate optical flow fields are produced for a given in-between frame, one predicted from our network and the other estimated from those of neighboring frames using a flow fusion map. We also employ an image fusion map to combat occlusion problems in the warping processes, producing two candidate interpolated images that are fed to a shallow network with a residual structure to obtain the final interpolated image. To handle challenging scenarios, we apply a set of Gabor filters to extract phase variations in the feature domain with a multiscale phase subnetwork. Our entire neural network is end-to-end trainable. Our experiments show that this method outperforms the state-of-the-art approaches and achieves marked visual improvement in challenging scenarios.