Local Transformer With Spatial Partition Restore for Hyperspectral Image Classification

Abstract

Convolutional neural network (CNN) has exhibited enormous potentials in hyperspectral image (HSI) classification owing to excellent locally modeling ability. Although excellent performance of CNN-based methods has been witnessed, there still have some limitations of their internal network backbone. On the one hand, modeling long-distance context dependencies is an inborn defect, which leads to receptive field limitation and insufficient feature capture in HSI. On the other hand, CNN-based methods usually need various sample distribution to train and cannot infer dynamically, which may not capture the inherent changes of HSI data well. To overcome the above issues, we propose a novel local transformer with spatial partition restore network (SPRLT-Net) for HSI classification. Firstly, local transformer is introduced to obtain the spatial attention weights dynamically by measuring the similarity between related pixel pairs. Secondly, a spatial partition restore (SPR) module is designed to split the input patch into several overlapping continuous sub-patches as sequential. With the obtained attention weights at hand, the SPR module restores the sequential to the original patch. Finally, a fully connected layer is used for classification. SPRLT-Net can capture global context dependencies, and the dynamical attention weights can adapt the inherent changes of HSI spatial pixels. Experimental results based on spatially disjoint samples and randomly selected samples of five benchmark data sets demonstrate that SPRLT-Net outperforms the other state-of-the-art methods in terms of classification accuracy, generalization performance, and computational complexity.