A multi-range spectral-spatial transformer for hyperspectral image classification

Abstract

In recent years, the transformer-based approach becoming a hot research topic in hyperspectral image (HSI) classification tasks. However, most of these studies have focused on optimizing the model framework in pursuit of high-accuracy classification, with little attention to the composition of the input token sequence as an important factor affecting the performance of the transformer. Therefore, this paper further explores a novel token structure to strengthen the Transformer's performance for HSI classification tasks, based on which a Multi-Range Spectral-Spatial Transformer (MRSST) framework is developed. Specifically, a convolutional feature pre-encoder with two branches is designed to extract shallow features for each spectral channel separately. Then, a token generator is introduced to combine the shallow features with the raw spectral information to yield the token sequences with multi-range information. Finally, the tokens are input into the transformer encoders enhanced by a module that strengthens the information exchange between its mid-range and short-range semantic features. Experiments conducted on three well-known hyperspectral datasets demonstrate that the proposed multi-range composite token sequences and information exchange mechanisms significantly enhance the transformer's performance. Codes are released at: https://github.com/HyperSystemAndImageProc/Multi-Range-Spectral-Spatial-Transformer.