Two-Branch Pure Transformer for Hyperspectral Image Classification

Abstract

Owing to its capability of building long-range dependencies and global context connections, Transformer has been used for hyperspectral image (HSI) classification. However, most of the existing Transformer-based HSI spatial–spectral classification methods consist of a convolutional neural network (CNN) and Transformer, which are used to extract the local and global information, respectively. In this study, to fully explore the potential of Transformer, a pure Transformer is investigated for HSI classification. First, a spatial Transformer (Spa-TR) is designed for HSI spatial classification, which learns the spatial features locally and globally by adopting the window partition and shifted window schemes. Especially, the self-attention computations are limited within the local windows and cross-windows. Second, to fully use the abundant spectral information in HSIs, a two-branch pure Transformer (i.e., Spa-Spe-TR) is proposed, which includes a spectral Transformer (Spe-TR) and a Spa-TR. The spectral sequence features learned by Spe-TR and the spatial features generated by Spa-TR are effectively fused with a branch fusion strategy, which explicitly and automatically measures the importance between the joint spatial–spectral features and improves the discriminability of the joint features. Experimental results on the two widely used HSI datasets (i.e., Pavia and Indian Pines) demonstrate the efficacy of the proposed methods in comparison with other state-of-the-art approaches.