Spatial Spectral Joint Correction Network for Hyperspectral and Multispectral Image Fusion

Abstract

AbstractHyperspectral and multispectral image (HS-MSI) fusion aims to generate a high spatial resolution hyperspectral image (HR-HSI), using the complementarity and redundancy of the low spatial resolution hyperspectral image (LR-HSI) and the high spatial resolution multispectral image (HS-MSI). Previous works usually assume that the spatial down-sampling operator between HR-HSI and LR-HSI, and the spectral response function between HR-HSI and HR-MSI are known, which is infeasible in many cases. In this paper, we propose a coarse-to-fine HS-MSI fusion network, which does not require the prior on the mapping relationship between HR-HSI and LRI or MSI. Besides, the result is improved by iterating the proposed structure. Our model is composed of three blocks: degradation block, error map fusion block and reconstruction block. The degradation block is designed to simulate the spatial and spectral down-sampling process of hyperspectral images. Then, error maps in space and spectral domain are acquired by subtracting the degradation results from the inputs. The error map fusion block fuses those errors to obtain specific error maps corresponding to initialize HSI. In the case that the learned degradation process could represent the real mapping function, this block ensures to generate accurate errors between degraded images and the ground truth. The reconstruction block uses the fused maps to correct HSI, and finally produce high-precision hyperspectral images. Experiment results on CAVE and Harvard dataset indicate that the proposed method achieves good performance both visually and quantitatively compared with some SOTA methods.KeywordsHyperspectral imageImage fusionDeep learningDegradation model