Efficient SpectralFormer for hyperspectral image classification

Abstract

Convolutional neural network (CNN) and its variants have been widely applied to hyperspectral classification for their excellent ability to extract local features. However, as research on hyperspectral imaging (HSI) has progressed, CNNs have been proven to struggle in extracting and representing the sequential properties of spectral characteristics. Recently, some researchers have demonstrated the feasibility of transformer architecture in HSI classification due to its powerful ability to characterize spectral information. The lack of suitable pre-processing and optimization methods which are used for the transformer's application in HSI becomes a major limitation to model's performance. Therefore, to enhance model's performance and practicality, we propose an efficient transformer backbone for HSI classification, named Efficient-Spectralformer. In the proposed framework, we rethink the input form and design the Split Grouping Embedding (SGE) module that requires much less computational resources. Additionally, to maximize the use of attentional feature information from different layers, we have used the Multi-layer Feature Fusion (MFF) module is used to learn multi-layer spectral attention information. The experiments conducted on five HSI datasets demonstrate that the proposed method achieves better performance under much less memory usage (about 30% of the original on average) by comparing with the original Spectralformer.