Spectral-spatial Feature Extraction Research Articles

A Joint Network of Edge-Aware and Spectral–Spatial Feature Learning for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is a vital part of the HSI application field. Since HSIs contain rich spectral information, it is a major challenge to effectively extract deep representation features. In existing methods, although edge data augmentation is used to strengthen the edge representation, a large amount of high-frequency noise is also introduced at the edges. In addition, the importance of different spectra for classification decisions has not been emphasized. Responding to the above challenges, we propose an edge-aware and spectral–spatial feature learning network (ESSN). ESSN contains an edge feature augment block and a spectral–spatial feature extraction block. Firstly, in the edge feature augment block, the edges of the image are sensed, and the edge features of different spectral bands are adaptively strengthened. Then, in the spectral–spatial feature extraction block, the weights of different spectra are adaptively adjusted, and more comprehensive depth representation features are extracted on this basis. Extensive experiments on three publicly available hyperspectral datasets have been conducted, and the experimental results indicate that the proposed method has higher accuracy and immunity to interference compared to state-of-the-art (SOTA) method.

An improved 3D-SwinT-CNN network to evaluate the fermentation degree of black tea

Fermentation is a key process in forming the flavor quality of black tea. Evaluating the degree of fermentation during black tea processing is difficult. This paper proposes an improved 3D-SwinT-CNN network for the end-to-end processing of black tea’s hyperspectral images to evaluate the degree of fermentation. The model incorporates dilated convolution and a shifted window self-attention mechanism, expanding the network’s receptive field and capturing both local spatial-spectral and global features of black tea hyperspectral images. The accuracy is 98.13% on the test set. Compared to manual feature extraction methods, the accuracy improved by 3.45%. Compared to baseline 3D-CNNs, 3D-SwinT-CNN demonstrates superior spatial-spectral feature extraction, achieving an average accuracy improvement of 10.17%. Ablation experiments were conducted to further verify the effectiveness of the introduced modules. The proposed 3D-SwinT-CNN aims to establish a research foundation for online evaluation of black tea's fermentation degree.

MSTSENet: Multiscale Spectral–Spatial Transformer with Squeeze and Excitation network for hyperspectral image classification

Hyperspectral image (HSI) classification pertains to the task of assigning a single label to each pixel by analyzing its spectral–spatial characteristics. Convolutional Neural Networks (CNNs) have garnered significant attention on account of their remarkable performance in feature representation. However, these approaches possess a restricted capacity to acquire highly similar spectral features. Recently, Attention-based approaches have been devised as a means to surmount the limitations inherent in CNNs. In this paper, we address a novel architecture for spectral–spatial feature extraction and HSI classification. The framework efficiently amalgamates the robustness and efficacy of three modern backbone networks, i.e., Multiscale CNN, Squeeze and Excitation (SE), and Transformer named MSTSENet. Initially, the multiscale convolution module integrates three branches of 3D convolution layers, each employing distinct kernel sizes, facilitating multiscale and multi-resolution feature extraction. Subsequently, the SE module enhances the inter-channel relationship by adaptively recalibrating the feature weight. The strategic fusion of the SE module with multiscale CNN strengthens deep feature extraction while introducing a minimal increment in overall parameters. Lastly, the Transformer module orchestrates information aggregation across different spectral–spatial regions, accurately modeling long-range contextual dependencies and capturing dominant feature representations. To evaluate the model performance, comprehensive experiments have been carried out on widely used benchmark HSI datasets. The overall accuracy of 98.74%, 99.83%, and 99.98% is achieved over the Indian Pines, Pavia University, and Salinas Valley datasets, respectively. The code of this work will be accessible at https://github.com/irfan01000/MSTSENet.

Discriminating Spectral-Spatial Feature Extraction for Hyperspectral Image Classification: A Review.

Hyperspectral images (HSIs) contain subtle spectral details and rich spatial contextures of land cover that benefit from developments in spectral imaging and space technology. The classification of HSIs, which aims to allocate an optimal label for each pixel, has broad prospects in the field of remote sensing. However, due to the redundancy between bands and complex spatial structures, the effectiveness of the shallow spectral-spatial features extracted by traditional machine-learning-based methods tends to be unsatisfying. Over recent decades, various methods based on deep learning in the field of computer vision have been proposed to allow for the discrimination of spectral-spatial representations for classification. In this article, the crucial factors to discriminate spectral-spatial features are systematically summarized from the perspectives of feature extraction and feature optimization. For feature extraction, techniques to ensure the discrimination of spectral features, spatial features, and spectral-spatial features are illustrated based on the characteristics of hyperspectral data and the architecture of models. For feature optimization, techniques to adjust the feature distances between classes in the classification space are introduced in detail. Finally, the characteristics and limitations of these techniques and future challenges in facilitating the discrimination of features for HSI classification are also discussed further.

Transfer Learning-Based Hyperspectral Image Classification Using Residual Dense Connection Networks.

The extraction of effective classification features from high-dimensional hyperspectral images, impeded by the scarcity of labeled samples and uneven sample distribution, represents a formidable challenge within hyperspectral image classification. Traditional few-shot learning methods confront the dual dilemma of limited annotated samples and the necessity for deeper, more effective features from complex hyperspectral data, often resulting in suboptimal outcomes. The prohibitive cost of sample annotation further exacerbates the challenge, making it difficult to rely on a scant number of annotated samples for effective feature extraction. Prevailing high-accuracy algorithms require abundant annotated samples and falter in deriving deep, discriminative features from limited data, compromising classification performance for complex substances. This paper advocates for an integration of advanced spectral-spatial feature extraction with meta-transfer learning to address the classification of hyperspectral signals amidst insufficient labeled samples. Initially trained on a source domain dataset with ample labels, the model undergoes transference to a target domain with minimal samples, utilizing dense connection blocks and tree-dimensional convolutional residual connections to enhance feature extraction and maximize spatial and spectral information retrieval. This approach, validated on three diverse hyperspectral datasets-IP, UP, and Salinas-significantly surpasses existing classification algorithms and small-sample techniques in accuracy, demonstrating its applicability to high-dimensional signal classification under label constraints.

Classification of Microscopic Hyperspectral Images of Blood Cells Based on Lightweight Convolutional Neural Network

Hyperspectral imaging has emerged as a novel imaging modality in the medical field, offering the ability to acquire images of biological tissues while simultaneously providing biochemical insights for in-depth tissue analysis. This approach facilitates early disease diagnosis, presenting advantages over traditional medical imaging techniques. Addressing challenges such as the computational burden of existing convolutional neural networks (CNNs) and imbalances in sample data, this paper introduces a lightweight GhostMRNet for the classification of microscopic hyperspectral images of human blood cells. The proposed model employs Ghost Modules to replace conventional convolutional layers and a cascading approach with small convolutional kernels for multiscale feature extraction, aiming to enhance feature extraction capabilities while reducing computational complexity. Additionally, an SE (Squeeze-and-Excitation) module is introduced to selectively allocate weights to features in each channel, emphasizing informative features and efficiently achieving spatial–spectral feature extraction in microscopic hyperspectral imaging. We evaluated the performance of the proposed GhostMRNet and compared it with other state-of-the-art models using two real medical hyperspectral image datasets. The experimental results demonstrate that GhostMRNet exhibits a superior performance, with an overall accuracy (OA), average accuracy (AA), and Kappa coefficient reaching 99.965%, 99.565%, and 0.9925, respectively. In conclusion, the proposed GhostMRNet achieves a superior classification performance at a smaller computational cost, thereby providing a novel approach for blood cell detection.

Multi-Scale Residual Spectral–Spatial Attention Combined with Improved Transformer for Hyperspectral Image Classification

Aiming to solve the problems of different spectral bands and spatial pixels contributing differently to hyperspectral image (HSI) classification, and sparse connectivity restricting the convolutional neural network to a globally dependent capture, we propose a HSI classification model combined with multi-scale residual spectral–spatial attention and an improved transformer in this paper. First, in order to efficiently highlight discriminative spectral–spatial information, we propose a multi-scale residual spectral–spatial feature extraction module that preserves the multi-scale information in a two-layer cascade structure, and the spectral–spatial features are refined by residual spectral–spatial attention for the feature-learning stage. In addition, to further capture the sequential spectral relationships, we combine the advantages of Cross-Attention and Re-Attention to alleviate computational burden and attention collapse issues, and propose the Cross-Re-Attention mechanism to achieve an improved transformer, which can efficiently alleviate the heavy memory footprint and huge computational burden of the model. The experimental results show that the overall accuracy of the proposed model in this paper can reach 98.71%, 99.33%, and 99.72% for Indiana Pines, Kennedy Space Center, and XuZhou datasets, respectively. The proposed method was verified to have high accuracy and effectiveness compared to the state-of-the-art models, which shows that the concept of the hybrid architecture opens a new window for HSI classification.

A two-branch multiscale spectral-spatial feature extraction network for hyperspectral image classification

In the field of hyperspectral image (HSI) classification in remote sensing, the combination of spectral and spatial features has gained considerable attention. In addition, the multiscale feature extraction approach is very effective at improving the classification accuracy for HSIs, capable of capturing a large amount of intrinsic information. However, some existing methods for extracting spectral and spatial features can only generate low-level features and consider limited scales, leading to low classification results, and dense-connection based methods enhance the feature propagation at the cost of high model complexity. This paper presents a two-branch multiscale spectral-spatial feature extraction network (TBMSSN) for HSI classification. We design the multiscale spectral feature extraction (MSEFE) and multiscale spatial feature extraction (MSAFE) modules to improve the feature representation, and a spatial attention mechanism is applied in the MSAFE module to reduce redundant information and enhance the representation of spatial features at multiscale. Then we densely connect series of MSEFE or MSAFE modules respectively in a two-branch framework to balance efficiency and effectiveness, alleviate the vanishing-gradient problem and strengthen the feature propagation. To evaluate the effectiveness of the proposed method, the experimental results were carried out on bench mark HSI datasets, demonstrating that TBMSSN obtained higher classification accuracy compared with several state-of-the-art methods.

Hyperspectral Image Classification Using Multi-Scale Lightweight Transformer

The distinctive feature of hyperspectral images (HSIs) is their large number of spectral bands, which allows us to identify categories of ground objects by capturing discrepancies in spectral information. Convolutional neural networks (CNN) with attention modules effectively improve the classification accuracy of HSI. However, CNNs are not successful in capturing long-range spectral–spatial dependence. In recent years, Vision Transformer (VIT) has received widespread attention due to its excellent performance in acquiring long-range features. However, it requires calculating the pairwise correlation between token embeddings and has the complexity of the square of the number of tokens, which leads to an increase in the computational complexity of the network. In order to cope with this issue, this paper proposes a multi-scale spectral–spatial attention network with frequency-domain lightweight Transformer (MSA-LWFormer) for HSI classification. This method synergistically integrates CNN, attention mechanisms, and Transformer into the spectral–spatial feature extraction module and frequency-domain fused classification module. Specifically, the spectral–spatial feature extraction module employs a multi-scale 2D-CNN with multi-scale spectral attention (MS-SA) to extract the shallow spectral–spatial features and capture the long-range spectral dependence. In addition, The frequency-domain fused classification module designs a frequency-domain lightweight Transformer that employs the Fast Fourier Transform (FFT) to convert features from the spatial domain to the frequency domain, effectively extracting global information and significantly reducing the time complexity of the network. Experiments on three classic hyperspectral datasets show that MSA-LWFormer has excellent performance.

Effective hyperspectral image classification based on segmented PCA and 3D-2D CNN leveraging multibranch feature fusion

ABSTRACT We present an innovative hyperspectral image (HSI) classification method addressing challenges posed by closely spaced wavelength bands. Our approach combines 3D-2D convolutional neural networks (CNNs) with multi-branch feature fusion for improved spectral-spatial feature extraction. Using segmented principal component analysis (Seg-PCA), we reduce HSIs’ spectral dimensions into global and local intrinsic characteristics. The integration of 3D and 2D CNNs captures joint spectral-spatial features, while a multi-branch network extracts and merges diverse local features along the spectral dimension. Our method outperforms existing approaches, achieving remarkable accuracy of 99.27%, 100%, and 99.99% on Indian Pines, Salinas Scene, and University of Pavia datasets, respectively.

Hyperspectral Image Super-Resolution Based on Feature Diversity Extraction

Deep learning is an important research topic in the field of image super-resolution. Problematically, the performance of existing hyperspectral image super-resolution networks is limited by feature learning for hyperspectral images. Nevertheless, the current algorithms exhibit some limitations in extracting diverse features. In this paper, we address limitations to existing hyperspectral image super-resolution networks, focusing on feature learning challenges. We introduce the Channel-Attention-Based Spatial–Spectral Feature Extraction network (CSSFENet) to enhance hyperspectral image feature diversity and optimize network loss functions. Our contributions include: (a) a convolutional neural network super-resolution algorithm incorporating diverse feature extraction to enhance the network’s diversity feature learning by elevating the matrix rank, (b) a three-dimensional (3D) feature extraction convolution module, the Channel-Attention-Based Spatial–Spectral Feature Extraction Module (CSSFEM), to boost the network’s performance in both the spatial and spectral domains, (c) a feature diversity loss function designed based on the image matrix’s singular value to maximize element independence, and (d) a spatial–spectral gradient loss function introduced based on space and spectrum gradient values to enhance the reconstructed image’s spatial–spectral smoothness. In contrast to existing hyperspectral super-resolution algorithms, we used four evaluation indexes, PSNR, mPSNR, SSIM, and SAM, and our method showed superiority during testing with three common hyperspectral datasets.

Fast and Accurate Hyperspectral Image Classification with Window Shape Adaptive Singular Spectrum Analysis

Hyperspectral classification is a task of significant importance in the field of remote sensing image processing, with attaining high precision and rapid classification increasingly becoming a research focus. The classification accuracy depends on the degree of raw HSI feature extraction, and the use of endless classification methods has led to an increase in computational complexity. To achieve high accuracy and fast classification, this study analyzes the inherent features of HSI and proposes a novel spectral–spatial feature extraction method called window shape adaptive singular spectrum analysis (WSA-SSA) to reduce the computational complexity of feature extraction. This method combines similar pixels in the neighborhood to reconstruct every pixel in the window, and the main steps are as follows: rearranging the spectral vectors in the irregularly shaped region, constructing an extended trajectory matrix, and extracting the local spatial and spectral information while removing the noise. The results indicate that, given the small sample sizes in the Indian Pines dataset, the Pavia University dataset, and the Salinas dataset, the proposed algorithm achieves classification accuracies of 97.56%, 98.34%, and 99.77%, respectively. The classification speed is more than ten times better than that of other methods, and a classification time of only about 1–2 s is needed.

Spectral–Spatial Feature Extraction for Hyperspectral Image Classification Using Enhanced Transformer with Large-Kernel Attention

In the hyperspectral image (HSI) classification task, every HSI pixel is labeled as a specific land cover category. Although convolutional neural network (CNN)-based HSI classification methods have made significant progress in enhancing classification performance in recent years, they still have limitations in acquiring deep semantic features and face the challenges of escalating computational costs with increasing network depth. In contrast, the Transformer framework excels in expressing high-level semantic features. This study introduces a novel classification network by extracting spectral–spatial features with an enhanced Transformer with Large-Kernel Attention (ETLKA). Specifically, it utilizes distinct branches of three-dimensional and two-dimensional convolutional layers to extract more diverse shallow spectral–spatial features. Additionally, a Large-Kernel Attention mechanism is incorporated and applied before the Transformer encoder to enhance feature extraction, augment comprehension of input data, reduce the impact of redundant information, and enhance the model’s robustness. Subsequently, the obtained features are input to the Transformer encoder module for feature representation and learning. Finally, a linear layer is employed to identify the first learnable token for sample label acquisition. Empirical validation confirms the outstanding classification performance of ETLKA, surpassing several advanced techniques currently in use. This research provides a robust and academically rigorous solution for HSI classification tasks, promising significant contributions in practical applications.

Hyperspectral Image Classification Network Based on 3D Octave Convolution and Multiscale Depthwise Separable Convolution

Hyperspectral images (HSIs) are pivotal in various fields due to their rich spectral–spatial information. While convolutional neural networks (CNNs) have notably enhanced HSI classification, they often generate redundant spatial features. To address this, we introduce a novel HSI classification method, OMDSC, employing 3D Octave convolution combined with multiscale depthwise separable convolutional networks. This method initially utilizes 3D Octave convolution for efficient spectral–spatial feature extraction from HSIs, thereby reducing spatial redundancy. Subsequently, multiscale depthwise separable convolution is used to further improve the extraction of spatial features. Finally, the HSI classification results are output by softmax classifier. This work compares the method with other methods on three publicly available datasets in order to confirm its efficacy. The outcomes show that the method performs better in terms of classification.

Unleashing the full potential of hyperspectral imaging: Decoupled image and frequency-domain spatial–spectral framework

Hyperspectral image classification (HSIC) is a rapidly developing field that utilizes deep learning methods. However, the reliance on convolutional neural networks (CNNs) for spectral–spatial feature extraction presents certain limitations. Specifically, the use of the fixed-position convolutional kernels in CNNs hinders their ability to capture fully the spectral information around the spatial central pixel, thereby overlooking critical differences between features. To address this issue, a decoupled image- and frequency-domain spectral–spatial framework for HSIC was developed in this study. This method incorporates image- and frequency-domain-based multiscale learnable convolutional attention to refine the differentiating features of the different feature distributions. Additionally, a novel frequency-domain information enhancement module was designed to extract the structural shape and texture details under the semantic constraints of the frequency phase, complementing the image domain to improve the extracted feature maps. Furthermore, a simple and efficient hierarchical feature representation module was introduced to extract both local and global information effectively from the fused features. The experimental results obtained using three open datasets and a practical hyperspectral image of the Gaofen-5 satellite demonstrate that the proposed method outperforms other state-of-the-art HSIC methods.

TransHSI: A Hybrid CNN-Transformer Method for Disjoint Sample-Based Hyperspectral Image Classification

Hyperspectral images’ (HSIs) classification research has seen significant progress with the use of convolutional neural networks (CNNs) and Transformer blocks. However, these studies primarily incorporated Transformer blocks at the end of their network architectures. Due to significant differences between the spectral and spatial features in HSIs, the extraction of both global and local spectral–spatial features remains incomplete. To address this challenge, this paper introduces a novel method called TransHSI. This method incorporates a new spectral–spatial feature extraction module that leverages 3D CNNs to fuse Transformer to extract the local and global spectral features of HSIs, then combining 2D CNNs and Transformer to capture the local and global spatial features of HSIs comprehensively. Furthermore, a fusion module is proposed, which not only integrates the learned shallow and deep features of HSIs but also applies a semantic tokenizer to transform the fused features, enhancing the discriminative power of the features. This paper conducts experiments on three public datasets: Indian Pines, Pavia University, and Data Fusion Contest 2018. The training and test sets are selected based on a disjoint sampling strategy. We perform a comparative analysis with 11 traditional and advanced HSI classification algorithms. The experimental results demonstrate that the proposed method, TransHSI algorithm, achieves the highest overall accuracies and kappa coefficients, indicating a competitive performance.

Central Attention Network for Hyperspectral Imagery Classification.

In this article, the intrinsic properties of hyperspectral imagery (HSI) are analyzed, and two principles for spectral-spatial feature extraction of HSI are built, including the foundation of pixel-level HSI classification and the definition of spatial information. Based on the two principles, scaled dot-product central attention (SDPCA) tailored for HSI is designed to extract spectral-spatial information from a central pixel (i.e., a query pixel to be classified) and pixels that are similar to the central pixel on an HSI patch. Then, employed with the HSI-tailored SDPCA module, a central attention network (CAN) is proposed by combining HSI-tailored dense connections of the features of the hidden layers and the spectral information of the query pixel. MiniCAN as a simplified version of CAN is also investigated. Superior classification performance of CAN and miniCAN on three datasets of different scenarios demonstrates their effectiveness and benefits compared with state-of-the-art methods.

LG-3D-SSA: Local and Global Three Dimensional SSA to Achieve Efficient Spectral-Spatial Feature Extraction of Hyperspectral Images

LG-3D-SSA: Local and Global Three Dimensional SSA to Achieve Efficient Spectral-Spatial Feature Extraction of Hyperspectral Images

A Decompressed Spectral-Spatial Multiscale Semantic Feature Network for Hyperspectral Image Classification

Convolutional neural networks (CNNs) have shown outstanding feature extraction capability and become a hot topic in the field of hyperspectral image (HSI) classification. However, most of the prior works usually focus on designing deeper or wider network architectures to extract spatial and spectral features, which give rise to difficulty for optimization and more parameters along with higher computation. Moreover, how to learn spatial and spectral information more effectively is still being researched. To tackle the aforementioned problems, a decompressed spectral-spatial multiscale semantic feature network (DSMSFNet) for HSI classification is proposed. This model is composed of a decompressed spectral-spatial feature extraction module (DSFEM) and a multiscale semantic feature extraction module (MSFEM). The former is devised to extract more discriminative and representative global decompressed spectral-spatial features in a lightweight extraction manner, while the latter is constructed to expand the range of available receptive fields and generate clean multiscale semantic features at a granular level to further enhance the classification performance. Compared with progressive classification approaches, abundant experimental results on three benchmark datasets prove the superiority of our developed DSMSFNet model.

Dual-branch spectral–spatial feature extraction network for multispectral image compression

Dual-branch spectral–spatial feature extraction network for multispectral image compression