A Cross-Level Spectral–Spatial Joint Encode Learning Framework for Imbalanced Hyperspectral Image Classification

Abstract

Convolutional neural networks (CNNs) have dominated the research of hyperspectral image (HSI) classification, attributing to the superior feature representation capacity. Patch-free global learning (FPGA) as a fast learning framework for HSI classification has received wide interest. Despite their promising results from the perspective of fast inference, recent works have difficulty modeling spectral-spatial relationships with imbalanced samples. In this paper, we revisit the encoder–decoder-based fully convolutional network (FCN) and propose a cross-level spectral-spatial joint encoding framework (CLSJE) for Imbalanced HSI classification. First, a multi-scale input encoder and multiple-to-one multi-scale features connection are introduced to obtain abundant features and facilitate multi-scale contextual information flow between encoder and decoder. Second, in the encoder layer, we propose the spectral-spatial joint attention (SSJA) mechanism consisting of the high-frequency spatial attention (HFSA) and spectral-transform channel attention (STCA). HFSA and STCA encode spectral-spatial features jointly to improve the learning of the discriminative spectral-spatial features. Powered by these two components, CLSJE enjoys a high capability to capture both spatial and spectral dependencies for HSI classification. Besides, a class-proportion sampling strategy is developed to increase the attention to insufficiency samples. Extensive experiments demonstrate the superiority of our proposed CLSJE both at classification accuracy and inference speed, and show the state-of-the-art results on four benchmark datasets. Code can be obtained at: https://github.com/yudadabing/CLSJE.