NonRegSRNet: A Nonrigid Registration Hyperspectral Super-Resolution Network

Abstract

Due to the limitations of imaging systems, satellite hyperspectral imagery (HSI), which yields rich spectral information in many channels, often suffers from poor spatial resolution. HSI super-resolution (SR) refers to the fusion of high spatial resolution multispectral imagery (MSI) and low spatial resolution HSI to generate HSI that has both a high spatial and high spectral resolution. However, most existing SR methods assume that the two original images used are perfectly registered: in reality, nonrigid deformation areas can exist locally in the two images even if prior registration of the control points has been carried out. To address this problem, we propose a novel unsupervised spectral unmixing and image deformation correction network&#x2014;NonRegSRNet&#x2014;with multimodal and multitask learning that can be used for the joint registration of HSI and MSI and to produce SR imagery. More specifically, NonRegSRNet integrates the dense registration and SR tasks into a unified model that includes a triplet convolutional neural network. This allows these two tasks to complement each other so that better registration and SR results can be achieved. Furthermore, because the point spread function (PSF) and spectral response function (SRF) are often unavailable, two special convolutional layers are designed to adaptively learn the parameters of the PSF and SRF, which makes the proposed model more adaptable. Experimental results demonstrate that the proposed method has the ability to produce highly accurate and stable reconstructed images under complex nonrigid deformation conditions. (Code available at <uri>https://github.com/saber-zero/NonRegSRNet</uri>)