Super Resolution for Multi Frames with 3D Feature Extraction and RNN Prediction

Abstract

Super-resolution research has received extensive attention in recent years. With the rise and maturity of deep learning technology, learning-based superresolution (SR) method has gradually become the mainstream. Its main idea is to directly learn the mapping between low-resolution images and highresolution images by constructing a corresponding neural network. Different from the SR method only for a single image, the study of video SR is developing at the same time. For videos, in addition to considering the spatial characteristics of each frame, the variation of between frames in the time dimension is also a vital element affecting the video SR performance. However, the use of inter-frame motion characteristics remains a problem. Due to the characteristic limitations of the convolutional neural networks (CNN), it is not an efficient way to model the dependency in time dimension. There are also some SR methods using the optical flow to help with the modeling, but they still struggle with a problem of coping with complex large-scale motions. In order to deal with the above problems, we propose a network utilizing 3D convolution and recurrent neural network (RNN). In the proposed method, we use 3D convolution to extract the intra-frame and inter-frame features, then combine it with our original RNN to model the long-term dependence and short-term dependence in the video sequence. Furthermore, for reducing the huge amount of computational cost caused by the 3D convolution, we used the structure of the Laplacian pyramid to balance the calculation of the network. Finally, we conducted an experimental evaluation of our model and confirmed its effectiveness.