S2Former: Parallel Spectral–Spatial Transformer for Hyperspectral Image Classification

Abstract

Due to their excellent representation talent in local features, the convolutional neural network (CNN) has achieved favourable performance in hyperspectral image (HSI) classification tasks. Nevertheless, current CNN models exhibit a marked flaw: they are hard to model the dependencies in long-range distanced positions. This flaw becomes more problematic for the HSI classification task, which targets extracting more discriminative features in local and global dimensions from limited samples. In this paper, we introduce a spatial–spectral transformer (S2Former), which explores spatial and spectral feature extraction in a dual-stream framework for HSI Classification. S2Former, which consists of a spatial transformer and a spectral transformer in parallel branches, extracts the discriminative feature in spatial and spectral dimensions. More specifically, we propose multi-head spatial self-attention to capture the long-range spatial dependency of non-adjacent HSI pixels in a spatial transformer. In the spectral transformer, we propose multi-head covariance spectral attention to mine and represent spectral signatures by computing covariance-based channel maps. Meanwhile, the local activation feed-forward network is developed to complement local details. Extensive experiments conducted on four publicly available datasets indicate that our S2Former achieves state-of-the-art performance for the HSI classification task.