Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion

Abstract

Hyperspectral image (HSI) fusion can effectively improve the spatial resolution of HSIs by integrating high-resolution multispectral images (MSIs). Considering the spatial and spectral degradation relationship between a fused image and input images, a physics-based GAN is proposed to fuse HSI and MSI. A physical model estimating degradation of image is introduced in the generator and in the discriminators. For the generator, a set of recursive modules including a physical degradation model and a multiscale residual channel attention fusion module integrate the spectral-spatial difference information between input images and estimated degradation images to restore the details of the fused image. Subsequently, the residual spatial attention fusion module is used to combine the results of all recursions to obtain the final reconstructed result. As for the discriminators, three networks with the final fused image, estimated LR HSI and estimated MSI as inputs share the same architecture. Finally, the loss function that contains adversarial losses and L1 losses of the fused image and estimated degradation images is used to optimize network parameters. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods.