Joint Image Super-Resolution Via Recurrent Convolutional Neural Networks With Coupled Sparse Priors

Abstract

Joint image super-resolution (SR) refers to the reconstruction of a high-resolution image from its low-resolution version with the aid of a high-resolution image from another modality. Inspired by the recent success of recurrent neural networks in single image SR, we propose a novel multimodal recurrent convolutional neural network with coupled sparse priors for joint image SR. Our network fuses representations of the two image modalities at input layers using a learned multimodal convolutional sparse coding network. Additional recurrent convolutional stages are performed to further learn the mapping between the input modalities and the desired highresolution estimate. We apply the proposed network to the tasks of near-infrared image SR and multi-spectral image SR using RGB images as the guidance modality. Experimental results show the superior performance of the proposed multimodal recurrent convolutional network against several state-of-the-art single-modal and multimodal image SR methods.