Kernel adaptive memory network for blind video super-resolution

Abstract

Although recent video super-resolution (VSR) works show remarkable restoration performance for low-resolution (LR) video downscaled by a fixed known blur kernel, blind VSR suffers from severe performance degradation when the blur kernel is unknown. To alleviate this problem, blur kernel estimation methods have been proposed for VSR. However, existing VSR models must be trained separately with each LR dataset downscaled using all possible blur kernels. This is a time-consuming and memory-consuming task. To address these issues, we propose a kernel adaptive memory network for a blind VSR (KeMoVSR). The KeMoVSR mainly consists of a dual regression blur kernel estimator and a kernel adaptive VSR. The proposed blur kernel estimator predicts the parametric and non-parametric blur kernels by exploiting the blur kernel variation. Due to the blur kernel variation, the proposed kernel estimator can consider the temporal consistency of the blur kernel variation in adjacent frames, which leads to accurate blur kernel estimation in LR video frames. The proposed kernel adaptive VSR modulates the VSR weights according to the shape of the blur kernel by weight modulation layers. By integrating the proposed methods based on the memory network, we propose the KeMoVSR, which performs VSR by adaptively modulating the VSR weights using the blur kernel parameters as keys and values in memory networks. Experiments show that the KeMoVSR achieves superior performance compared to other blind VSR approaches. The KeMoVSR provides effective memory utilization that is appropriate for real-world scenarios. The code is available at https://github.com/dbseorms16/KeMoVSR.