Space-time super-resolution with motion-perceptive deformable alignment

Abstract

We proposed a motion-perceptive deformable alignment network that introduces a pre-computed optical flow to improve the motion perception of the deformable alignment process. The pre-computed flows share the burden of the learned offsets for motion estimation while the flexibility of the deformable convolutional network is maintained. In addition, a motion-adaptive pyramid structure in which the features are aligned in multi-scale levels and then merged based on the motion strength among input frames is proposed. With the above structures, an innovative space-time super-resolution (STSR) network is constructed with an improved motion compensation ability. STSR aims to restore a high-resolution and high-frame-rate sequence from its corresponding low-resolution and low-frame-rate version. The proposed STSR network is trained with a Vimeo-90K dataset, and tests are conducted on the Vimeo-90K, densely-annotated video segmentation (DAVIS), and realistic and diverse scenes (REDS) datasets. Performance is evaluated using the peak signal-to-noise ratio and structural similarity index of the entire restored frame on the Y channel. Extensive experiments demonstrate the superiority of the proposed network among both the one- and two-stage STSR methods, its improved alignment ability, and its significantly improved interpolated frame synthesis.