Transformer-enhanced two-stream complementary convolutional neural network for hyperspectral image classification

Abstract

Convolutional neural networks (CNNs) and Transformers have been widely employed in hyperspectral image (HSI) classification, due to their advantages in extracting local-range and global-range features respectively. However, some hybrid CNNs and Transformer-based methods still face some shortcomings in effectively exploiting local-global features in HSI. In this article, a Transformer-enhanced two-stream complementary convolutional neural network is proposed for HSI classification. It mainly comprises the spectral feature extraction stream, spatial feature extraction stream, and a spectral-spatial weight feature complementary module. Firstly, in spectral feature extraction stream, a hybrid convolution block is proposed to extract high-resolution spectral features. And then an attention mechanism, which is based on a Transformer encoder, is used to learn key informative features in spectral channel. The spatial feature extraction stream is designed for spatial feature extraction, and it has a similar architecture to the spectral feature extraction stream. Secondly, a spectral-spatial weight feature complementary module is developed to fully exploit the features which are based on the feature weight in spectral tokens and spatial tokens, they are from spectral and spatial feature extraction streams separately, and then the spectral-spatial tokens with different semantic features are derived. Finally, the spectral-spatial tokens are fed into the linear layer to produce classification outcomes. Furthermore, simplified convolution kernels are employed in our proposed method to decrease the total parameters. Results from experiments on four extensively used HSI datasets highlight the superior performance of our proposed method over other cutting-edge methods.