Multispectral Pansharpening Based on Multisequence Convolutional Recurrent Neural Network

Abstract

Multispectral (MS) pansharpening is defined as the fusion of spatial information in panchromatic (PAN) image and spectral information in MS image. In this work, we propose a MS pansharpening based on multi-sequence convolutional recurrent neural network (MCRNN). The proposed MCRNN contains two sub-networks (shallow feature extraction sub-network and deep feature fusion sub-network). In the shallow feature extraction sub-network, PAN and MS images are superimposed in the spectral dimension as multi-sequence data. A convolutional neural network (CNN) based on residual learning is then used to obtain the feature maps from multi-sequence data. In the deep feature fusion sub-network, since MS and PAN images are highly correlated, a convolutional recurrent neural network (ConvGRU) belonging to RNN is used to model adjacent and across-band relationships between these feature maps to capture the local and global correlations of the features in different bands. The global average pooling is then performed on the output results to yield the pansharpening result. Several datasets are tested at reduced and full resolution experiments, the experimental results show that the performance of the proposed MCRNN is superior to the traditional pansharpening methods. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HZC-1998/Multi-Sequence-Convolutional-Recurrent-Network-for-Pansharpening</uri> .