Multi-Scale CNN-Transformer Dual Network for Hyperspectral Compressive Snapshot Reconstruction

Abstract

Coded aperture snapshot spectral imaging (CASSI) is a new imaging mode that captures the spectral characteristics of materials in real scenes. It encodes three-dimensional spatial–spectral data into two-dimensional snapshot measurements, and then recovers the original hyperspectral image (HSI) through a reconstruction algorithm. Hyperspectral data have multi-scale coupling correlations in both spatial and spectral dimensions. Designing a network architecture that effectively represents this coupling correlation is crucial for enhancing reconstruction quality. Although the convolutional neural network (CNN) can effectively represent local details, it cannot capture long-range correlation well. The Transformer excels at representing long-range correlation within the local window, but there are also issues of over-smoothing and loss of details. In order to cope with these problems, this paper proposes a dual-branch CNN-Transformer complementary module (DualCT). Its CNN branch mainly focuses on learning the spatial details of hyperspectral images, and the Transformer branch captures the global correlation between spectral bands. These two branches are linked through bidirectional interactions to promote the effective fusion of spatial–spectral features of the two branches. By utilizing characteristics of CASSI imaging, the residual mask attention is also designed and encapsulated in the DualCT module to refine the fused features. Furthermore, by using the DualCT module as a basic component, a multi-scale encoding and decoding model is designed to capture multi-scale spatial–spectral features of hyperspectral images and achieve end-to-end reconstruction. Experiments show that the proposed network can effectively improve reconstruction quality, and ablation experiments also verify the effectiveness of our network design.