UVMSR: a novel approach to hyperspectral image super-resolution by fusing U-Net and Mamba

Abstract

Listen

Translate article

ABSTRACT The hyperspectral image super resolution (HSISR) task has been thoroughly researched and has shown notable advancements. However, existing deep neural network-based methods for HSISR face challenges in effectively utilizing global spectral-spatial information. While transformer-based models exhibit strong global modelling capabilities, their high computational complexity poses a challenge when applied to hyperspectral image processing. Recently, state space models (SSM) with efficient hardware-aware design, such as Mamba, have demonstrated promising capabilities for long sequence modelling. In this study, we introduce a HSISR method (UVMSR) that combines U-Net and Mamba. UVMSR is a hybrid CNN-SSM module that integrates the local feature extraction capabilities of convolutional layers with the long-range dependency capturing abilities of SSMs. Specifically, we design the U-Net network structure for HSISR and apply V-Mamba within it for global modelling to capture spectral-spatial features. V-Mamba utilizes positional embedding to label the image sequences and employs a bidirectional state-space model for global context modelling. Additionally, a spectral-spatial feature expansion (SSFE) module is designed for better recovery of detailed information in hyperspectral images during the up-sampling process of U-Net. This paper evaluates the performance of UVMSR on the Chikusei, Pavia Centre, Houston 2018 and Cave datasets. The results of the comparison with other state-of-the-art methods demonstrate that UVMSR outperforms them, achieving unparalleled performance in reconstruction results. The code is available at https://github.com/TeresaTing/UVMSR.