Coarse-to-fine SVD-GAN based framework for enhanced frame synthesis

Abstract

Frame interpolation and synthesis are growing topics in the field of computer vision. Hence, these topics gained more attention recently where several deep-learning architectures were proposed to enhance the quality of the synthesized frames. In this paper, an efficient handcrafted deep approach is proposed for better frame synthesis. The proposed approach takes advantage of singular value decomposition (SVD) framework and Generative Adversarial Networks (GAN). The proposed approach does not require any computationally expensive feature extraction steps such optical flow techniques and block-based motion compensation techniques. Nonetheless, the SVD components still carry the relevant motion information needed to deal with the challenges such as large motion and occlusion. Thus, the frames are temporally upscaled via SVD based construction procedure where new middle frames are interpolated for further enhancement using a GAN based approach that eliminates most of the visual artifacts. The proposed frame synthesis approach is comprehensively evaluated in different scenarios where its performance is assessed and compared with the state-of-the-art. Our framework outperforms the majority of the deep learning approaches in terms quantitative results in addition to qualitative results where our framework can generate smoother frames with less visual artifacts.