Deep Hyperspectral and Multispectral Image Fusion via Probabilistic Matrix Factorization

Abstract

Deep learning methods are popular for hyperspectral and multispectral image (HSI-MSI) fusion to obtain a high-resolution hyperspectral image. However, most of them are unsatisfactory due to limited generalization ability and poor interpretability. This paper proposes a highly-interpretable deep HSI-MSI fusion method based on Probabilistic Matrix Factorization (PMF) under the Bayesian framework. In the proposed method, a hyperspectral image is factorized into two matrices, namely, the Gaussian-prior-regularized spectral matrix and the deep-prior-regularized abundance matrix. Then, we split the optimization process into two meaningful iterative updating steps: updating the spectral matrix based on least squares estimation, and updating the abundance matrix based on a CNN-based Gaussian denoiser for 2D gray images. To improve the generalization ability, we provide solutions for selections of hyper-parameters, CNN-based denoiser architecture and training strategy. Using the given solutions, the proposed fusion method can be trained with 2D images once and then used to fuse different types of HSI and MSI excellently. Experiments on three datasets demonstrate that the proposed fusion method has good fusion performance and high generalization ability compared with other state-of-the-art methods. The source code will be available at https://github.com/KevinBHLin/.