DSS-TRM: deep spatial–spectral transformer for hyperspectral image classification

Abstract

ABSTRACT In recent years, the wide use of deep learning based methods has greatly improved the classification performance of hyperspectral image (HSI). As an effective method to improve the performance of deep convolution networks, attention mechanism is also widely used for HSI classification tasks. However, the majority of the existing attention mechanisms for HSI classification are based on the convolution layer, and the classification accuracy still has margins for improvement. Motivated by the latest self attention mechanism in natural language processing, a deep transformer is proposed for HSI classification in this paper. Specifically, deep transformer along the spectral dimension and the spatial dimension are explored respectively. Then, a deep spatial-spectral transformer (DSS-TRM) is proposed to improve the classification performance of HSI. The contribution of this paper is to make full use of self attention mechanism, that is to use transformer layer instead of convolution layer. More importantly, a DSS-TRM is proposed to realize end-to-end HSI classification. Extensive experiments are conducted on three HSI data sets. The experimental results demonstrates that the proposed DSS-TRM could outperform the traditional convolutional neural networks and attention based methods.