HiRN: Hierarchical recurrent neural network for video super-resolution (VSR) using two-stage feature evolution

Abstract

The aim of video super-resolution (VSR) is generate the high-resolution (HR) frames from their low-resolution (LR) counterparts. As one of the fundamental module of VSR, propagation process provides the path of feature map and specifies how the feature map is leveraged. In the recurrent propagation, the latent features can be propagated and aggregated. Therefore, adopting the recurrent strategy can resolve the limitation of sliding-window-based local propagation. Recently, bi-directional recurrent propagation-based latest methods have achieved powerful performance in VSR. However, existing bi-directional frameworks have structured by combining forward and backward branches. These structures cannot propagate and aggregate previous and future latent features of current branch. In this study, we suggest the hierarchical recurrent neural network (HiRN) based on feature evolution. The proposed HiRN is designed based on the hierarchical recurrent propagation and residual block-based backbone with temporal wavelet attention (TWA) module. The hierarchical recurrent propagation consists of two stages to combine advantages of low frame rate-based forward and backward schemes, and multi-frame rate-based bi-directional access structure. The proposed methods are compared with state-of-the-art (SOTA) methods on the benchmark datasets. Experiments show that the proposed scheme achieves superior performance compared with SOTA methods. In particular, the proposed HiRN achieves better performance than all compared methods in terms of SSIM on Vid4 benchmark. In addition, the proposed HiRN surpasses the existing GBR-WNN by a significant 3.03 dB in PSNR on REDS4 benchmark with fewer parameters.