Learning a Coupled Multilinear Network for Unsupervised Hyperspectral-Multispectral Image Fusion

Abstract

Fusing low resolution (LR) HSI with high resolution (HR) multispectral image (MSI) is an important technology to obtain HR hypersepctral image (HSI), which is hard to directly acquire due to the hardware limitation. Deep learning (DL) has been applied in HSI-MSI fusion, but the representability of DL networks for multidimensional (i.e., spectral-spatial) features still need improvement. And most DL HSI-MSI fusion networks are in supervised fashion, HR ground truth HSI is required for training, which is unavailable in reality. In this work, we investigate tensor theory, and propose a coupled multilinear network (CMuNet) for unsupervised HSI-MSI fusion, where deep image prior and degradation model can be jointly learned. CMuNet consists of coupled multilinear filtering subnets, it jointly represents the LR HSI and HR MSI as a random code and multidimensional features on spatial and spectral modes. The HR HSI is inferred with the random code, features on spatial modes of HR MSI and features on spectral mode of LR HSI. Experiments on several HSIs demonstrate the effectiveness of the proposed method.