HyperRefiner: a refined hyperspectral pansharpening network based on the autoencoder and self-attention

Abstract

ABSTRACT Deep learning has been developed to generate promising super resolution hyperspectral imagery by fusing hyperspectral imagery with the panchromatic band. However, it is still challenging to maintain edge spectral information in the necessary upsampling processes of these approaches, and difficult to guarantee effective feature extraction. This study proposes a pansharpening network denoted as HyperRefiner that consists of, (1) a well performing upsampling network SRNet, in which the dual attention block and refined attention block are cascaded to accomplish the extraction and fusion of features; (2) a spectral autoencoder that is embedded to perform dimensionality reduction under constrained feature extraction; and (3) the optimization module which performs self-attention at the pixel and feature levels. A comparison with several state-of-the-art models reveals that HyperRefiner can improve the quality of the fused image. Specifically, compared to the single-head HyperTransformer and with the Chikusei dataset, our network improved the Peak Signal-to-Noise Ratio, Erreur Relative Globale Adimensionnelle de Synthèse and Spectral Angle Mapper by 0.86%, 3.62%, and 2.09%, and reduce the total memory, floating point operations, model parameters and computation time by 41%, 75%, 86% and 46%, respectively. The experimental results show that HyperRefiner outperforms several networks and demonstrates its usefulness in hyperspectral image fusion. The code is publicly available at https://github.com/zsspo/Fusion_HyperRefiner.