Space-Time Video Super-Resolution 3D Transformer

Abstract

Space-time video super-resolution, which aims to generate a high resolution (HR) and high frame rate (HRF) video from a low frame rate (LFR), low resolution (LR) video. Simply combining video frame interpolation (VFI) and video super-resolution (VSR) network to solve this problem cannot bring satisfying performance, which also requires a heavy computational burden. In this paper, we investigate a one-stage network to jointly up-sample video both in time and space. In our framework, a 3D pyramid structure with channel attention is proposed to fuse input frames and generate intermediate features. The features are fed into the 3D Transformer network to model global relationships between features. Our proposed network, 3DTFSR, can efficiently process videos without explicit motion compensation. Extensive experiments on benchmark datasets demonstrate that the proposed method achieves better quantitative and qualitative performance compared to a two-stage network.