Integrating motion estimation and convolutional neural network for video super-resolution

Abstract

In this paper, we proposed a video super-resolution (SR) system. Image or video super-resolution has been studied for a long time. Recently, convolutional neural network (CNN) has been applied for image SR and provided impressive synthesized high-resolution results. Although CNN can provide better synthesized quality than traditional SR methods by preserving more high frequency information, the computational complexity is main concern for video super-resolution. In order to accelerate processing time, we integrated motion information into the SR system and then the SRCNN is used to reconstruct high resolution image. In other words, rather than reconstructing the whole frames for the input video, changed regions between two consecutive frames are explored and processed. Only unmatched patches were reconstructed via SRCNN in the proposed system. In addition, two additional strategies are proposed to speed up the processing time and reduce unexpected synthesized effects. According experimental results, the proposed system can save 37% computation time in one test video with dynamic changes.