Local Spectral Features Research Articles

Local-global feature fusion network for hyperspectral image classification

ABSTRACT Hyperspectral images (HSIs) are rich in spatial and spectral information, making them crucial for accurate classification. While existing convolutional neural network (CNN) methods have shown promise in HSI classification, they primarily focus on local features and struggle with establishing long-range dependencies. To address these limitations, we propose a Local-Global Feature Fusion Network (LGFFN). LGFFN effectively learns joint local-global information from HSIs. Firstly, we employ a group-general convolution parallel module (GGCPM) to extract both local and global spectral features. Next, we utilize parallel branches: a CNN branch for capturing fine-grained spatial information and an improved transformer encoder (ITE) branch for capturing long-distance spatial-spectral features and exploiting semantic information. Finally, we elegantly integrate the diverse features obtained from these branches to obtain classification results. We validate our method on four datasets and compare it with six state-of-the-art hyperspectral classification methods. Our experimental results demonstrate that LGFFN achieves high classification accuracy, outperforming existing methods.

Wave-PCT: Wavelet point cloud transformer for point cloud quality assessment

Point cloud representation for real-world objects has seen a surge in interest recently, finding widespread applications in augmented reality, virtual reality, and autonomous driving. However, the process of acquiring, compressing, transmitting, and processing point cloud data often introduces unwanted distortions, highlighting the need for effective no-reference quality assessment methods for distorted point clouds. In this paper, we propose a Wavelet Point Cloud Transformer (Wave-PCT) method to assess the quality of point clouds without reference data. We start by performing patch sampling on a source point cloud, generating an extensive set of patches from the point cloud. These point cloud patches and their 2D projection images are processed using wavelets to produce multiscale local spectral features; moreover, we also extract global features of these patches using Pointnet++. Finally, a self-attention module is performed to fuse these local and global features, resulting in a distortion-sensitive quality score metric. We evaluate the Wave-PCT method on several point cloud quality assessment datasets using various metrics, and the experimental results indicate that the proposed framework exhibits outstanding and reliable performance when compared to various point cloud quality assessment methods.

A frequency-domain approach with learnable filters for image classification

Machine learning applied to computer vision and signal processing is achieving results comparable to the human brain due to the great improvements brought by deep neural networks (DNN). The majority of state-of-the-art architectures are DNN-related, but only a few explicitly explore the frequency domain to extract useful information and improve the results. This paper presents a new approach for exploring the Fourier transform of the input images, which is composed of trainable frequency filters that boost discriminative components in the spectrum. Additionally, we propose a cropping procedure to allow the network to learn both global and local spectral features of the image blocks. The proposed method proved to be competitive concerning well-known DNN architectures in the selected experiments, which involved texture classification, cataract detection, and retina image analysis, where there is a noticeable appeal for the frequency domain, with the advantage of being a lightweight model.

Research on Convolutional Neural Network-Based Compression Methods for Multispectral Images

Considering the traits of multispectral images, which include numerous bands, high spatial and spectral redundancy, and a large data volume, there is a proposed investigation on compression methods based on convolutional neural networks to reduce the storage space consumed by an individual image and enhance compression effectiveness. This paper first introduces the development of image compression algorithms and deep learning in recent years. Based on these two structures, a framework for lossy compression of multispectral images utilizing an end-to-end convolutional neural network is proposed. A self-encoding structure is used to process three-dimensional hyperspectral images, extracting local spectral features and fusing spectral information using large convolutional kernels. Residual layers are employed to preserve spectral information. Rate-distortion optimization is performed to jointly optimize image distortion and compression bitrate. Finally, a comparison with the traditional JPEG method is conducted experiments to assess the efficacy of the proposed algorithm. The MS-SSIM is improved by nearly 0.08, and the compressed images exhibit no noticeable distortion.

Patch-Based Convolutional Encoder: A Deep Learning Algorithm for Spectral Classification Balancing the Local and Global Information.

Molecular vibrational spectroscopies, including infrared absorption and Raman scattering, provide molecular fingerprint information and are powerful tools for qualitative and quantitative analysis. They benefit from the recent development of deep-learning-based algorithms to improve the spectral, spatial, and temporal resolutions. Although a variety of deep-learning-based algorithms, including those to simultaneously extract the global and local spectral features, have been developed for spectral classification, the classification accuracy is still far from satisfactory when the difference becomes very subtle. Here, we developed a lightweight algorithm named patch-based convolutional encoder (PACE), which effectively improved the accuracy of spectral classification by extracting spectral features while balancing local and global information. The local information was captured well by segmenting the spectrum into patches with an appropriate patch size. The global information was extracted by constructing the correlation between different patches with depthwise separable convolutions. In the five open-source spectral data sets, PACE achieved a state-of-the-art performance. The more difficult the classification, the better the performance of PACE, compared with that of residual neural network (ResNet), vision transformer (ViT), and other commonly used deep learning algorithms. PACE helped improve the accuracy to 92.1% in Raman identification of pathogen-derived extracellular vesicles at different physiological states, which is much better than those of ResNet (85.1%) and ViT (86.0%). In general, the precise recognition and extraction of subtle differences offered by PACE are expected to facilitate vibrational spectroscopy to be a powerful tool toward revealing the relevant chemical reaction mechanisms in surface science or realizing the early diagnosis in life science.

Detection of intrinsic variants of an endmember in hyperspectral images based on local spatial and spectral features

Detection of intrinsic variants of an endmember in hyperspectral images based on local spatial and spectral features

TransHSI: A Hybrid CNN-Transformer Method for Disjoint Sample-Based Hyperspectral Image Classification

Hyperspectral images’ (HSIs) classification research has seen significant progress with the use of convolutional neural networks (CNNs) and Transformer blocks. However, these studies primarily incorporated Transformer blocks at the end of their network architectures. Due to significant differences between the spectral and spatial features in HSIs, the extraction of both global and local spectral–spatial features remains incomplete. To address this challenge, this paper introduces a novel method called TransHSI. This method incorporates a new spectral–spatial feature extraction module that leverages 3D CNNs to fuse Transformer to extract the local and global spectral features of HSIs, then combining 2D CNNs and Transformer to capture the local and global spatial features of HSIs comprehensively. Furthermore, a fusion module is proposed, which not only integrates the learned shallow and deep features of HSIs but also applies a semantic tokenizer to transform the fused features, enhancing the discriminative power of the features. This paper conducts experiments on three public datasets: Indian Pines, Pavia University, and Data Fusion Contest 2018. The training and test sets are selected based on a disjoint sampling strategy. We perform a comparative analysis with 11 traditional and advanced HSI classification algorithms. The experimental results demonstrate that the proposed method, TransHSI algorithm, achieves the highest overall accuracies and kappa coefficients, indicating a competitive performance.

Early detection of citrus anthracnose caused by Colletotrichum gloeosporioides using hyperspectral imaging

Citrus fruit are susceptible to Colletotrichum gloeosporioides infestation during postharvest and shelf storage. Early and accurate detection of citrus anthracnose is conducive for carrying out targeted pesticide control and mitigating the potential spread of the disease. An early citrus anthracnose detection method using hyperspectral imaging and machine learning techniques is proposed. The hyperspectral data of sound citrus fruits were first collected and served as healthy samples, which were then inoculated with C. gloeosporioides and were further divided into asymptomatic and symptomatic samples. To characterize the global and local grayscale differences of the susceptible samples in different band images, the mean spectrum of the region of interest (ROI) of each band image was extracted as the global spectral features; moreover, the ROI in each band was segmented into disjointed local regions using the contrast limited adaptive histogram equalization and the Otsu thresholding algorithm, where the mean spectrum of the local regions were extracted as the local spectral features, respectively. The global and local spectral features were then concatenated into fused spectral features. Finally, the performance of the fused features was evaluated using support vector machine (SVM), k-NN and random forest (RF). The results showed that, compared with the conventional spectral feature-based methods, the proposed fused spectral features combined with SVM obtained the optimal results, where the average detection accuracy reached 91.97%. Furthermore, after applying feature selection using the successive projections algorithm (SPA), the resultant dimensionally-reduced fused spectral features also obtained acceptable results, with an average detection accuracy of 91.04%.

Dual-Branch Fourier-Mixing Transformer Network for Hyperspectral Target Detection

In recent years, transformers have shown great potential in hyperspectral image processing and have also been gradually applied in hyperspectral target detection (HTD). Nonetheless, applying a typical transformer to HTD remains challenging. The heavy computation burden of the multi-head self-attention (MSA) in transformers limits its efficient HTD, while the limited ability to extract local spectral features can reduce the discrimination of the learned spectral features. To further explore the potential of transformers for HTD, for balance of representation ability and computational efficiency, we propose a dual-branch Fourier-mixing transformer network for hyperspectral target detection (DBFTTD). First, this work explores a dual-branch Fourier-mixing transformer network. The transformer-style network replaces the MSA sublayer in the transformer with a Fourier-mixing sublayer, which shows advantages in improving computational efficiency and learning valuable spectral information effectively for HTD. Second, this work proposes learnable filter ensembles in the Fourier domain that are inspired by ensemble learning to improve detection performance. Third, a simple but efficient dropout strategy is proposed for data augmentation. Sufficient and balanced training samples are constructed for training the dual-branch network, and training samples for balanced learning can further improve detection performance. Experiments on four data sets indicate that our proposed detector is superior to the state-of-the-art detectors.

Spectral-Spatial MLP Network for Hyperspectral Image Super-Resolution

Many hyperspectral image (HSI) super-resolution (SR) methods have been proposed and have achieved good results; however, they do not sufficiently preserve the spectral information. It is beneficial to sufficiently utilize the spectral correlation. In addition, most works super-resolve hyperspectral images using high computation complexity. To solve the above problems, a novel method based on a channel multilayer perceptron (CMLP) is presented in this article, which aims to obtain a better performance while reducing the computational cost. To sufficiently extract spectral features, a local-global spectral integration block is proposed, which consists of CMLP and some parameter-free operations. The block can extract local and global spectral features with low computational cost. In addition, a spatial feature group extraction block based on the CycleMLP framework is designed; it can extract local spatial features well and reduce the computation complexity and number of parameters. Extensive experiments demonstrate that our method achieves a good performance compared with other methods.

Effective infrared spectral identification method via improved local binary patterns and spectral description representation

Infrared spectral identification task which is an essential branch of infrared spectroscopy, has been extensively studied. In this article, we proposed an efficient description representation model with the local binary pattern (LBP) strategy, and applied to infrared spectral searching and identification. The idea is motivated by the fact that spectral curve can be considered as a composition of micro-patterns such as peaks, troughs, valley and flat lines which are well described by LBP. The whole spectral lines are split into several small sections, and then extracted the spectral LBP features. The method is capable of extracting both the local and holistic spectral features and robust to random noise and baseline interferences. Experimental results on the public infrared spectral dataset demonstrate that the proposed spectral identification method can significantly raise performance when noise and baseline interferences are present.

An advanced remote sensing retrieval method for urban non-optically active water quality parameters: An example from Shanghai

The optical insensitivity of non-optically active water quality parameters (NAWQPs) presents a significant challenge for remote sensing-based quantitative monitoring, which is an important tool for water quality assessment and management. Based on the analysis of the samples from Shanghai, China, it was found that the spectral morphological characteristics of the water body were obviously different under the combined effect of multiple NAWQPs. In view of this, in this paper, a machine learning method was proposed for the retrieval of urban NAWQPs by using multi-spectral scale morphological combined feature (MSMCF). The proposed method integrates both local and global spectral morphological features, and employs a multi-scale approach to enhance its applicability and stability, providing a more accurate and robust solution. To explore the applicability of the MSMCF method in retrieving urban NAWQPs, different methods were tested in terms of the retrieval accuracy and stability on the measured data and three different hyperspectral data. As can be seen from the results, the proposed method has good retrieval performance, which can be applied to hyperspectral data with different spectral resolutions with certain ability to suppress noise. Further analysis indicates that the sensitivity of each NAWQP to spectral morphological features varies. The research methods and findings in this paper can promote the development of hyperspectral and remote sensing technology in the prevention and treatment of urban water quality deterioration, and provide reference for related research.

Space Target Material Identification Based on Graph Convolutional Neural Network

Under complex illumination conditions, the spectral data distributions of a given material appear inconsistent in the hyperspectral images of the space target, making it difficult to achieve accurate material identification using only spectral features and local spatial features. Aiming at this problem, a material identification method based on an improved graph convolutional neural network is proposed. Superpixel segmentation is conducted on the hyperspectral images to build the multiscale joint topological graph of the space target global structure. Based on this, topological graphs containing the global spatial features and spectral features of each pixel are generated, and the pixel neighborhoods containing the local spatial features and spectral features are collected to form material identification datasets that include both of these. Then, the graph convolutional neural network (GCN) and the three-dimensional convolutional neural network (3-D CNN) are combined into one model using strategies of addition, element-wise multiplication, or concatenation, and the model is trained by the datasets to fuse and learn the three features. For the simulated data and the measured data, the overall accuracy of the proposed method can be kept at 85–90%, and their kappa coefficients remain around 0.8. This proves that the proposed method can improve the material identification performance under complex illumination conditions with high accuracy and strong robustness.

Multi-scale spectral-spatial dual-transformer network for hyperspectral image classification

ABSTRACT Deep learning methods have shown great advantages in hyperspectral image (HSI) classification tasks. In particular, convolutional neural network (CNN)-based methods for HSI classification have made great progress. However, it is difficult for CNN to process long-range spatial and spectral information. Recently, a novel deep learning model called transformer has demonstrated its potential to replace CNN in various classification tasks with its amazing performance. In this letter, a multi-scale spectral-spatial dual-transformer network (MS3DT) is proposed to deeply consider the spectral-spatial features via transformer. Specifically, MS3DT consists of a feature pyramid network (FPN), a spectral transformer subnetwork (SPECT) and a spatial transformer subnetwork (SPAT). To utilize the complementary multi-scale characteristics, we introduce FPN to capture shallow-to-deep and spectral-spatial features. To improve the representational capacity in spatial and spectral domains, SPECT is exploited to extract long-range spectral correlation over local spectral features, and SPAT is designed to explore the contextual information for better refinement. Therefore, MS3DT is able to adaptively recalibrate the nonlinear interdependence of shallow-to-deep spectral-spatial features by merging the two subnetworks. Experiments on two widely used HSI datasets show that the MS3DT can outperform the state-of-the-art (SOTA) algorithm. The source codes will be available at https://github.com/RsAI-lab/MSSSDT.

Global-Local-Feature-Fused Driver Speech Emotion Detection for Intelligent Cockpit in Automated Driving

Affective interaction between the intelligent cockpit and humans is becoming an emerging topic full of opportunities. Robust recognition of the driver's emotions is the first step for affective interaction, and the intelligent cockpit recognizes emotions through the driver's speech, which has a wide range of technical application potential. In this paper, we first proposed a multi-feature fusion parallel structure speech emotion recognition network, which complementarily fuses the global acoustic features and local spectral features of the entire speech. Second, we designed and conducted the speech data collection under the driver's emotion and established the driver's speech emotion (SpeechEmo) dataset in the dynamic driving environment including 40 participants. Finally, the proposed model was validated on the SpeechEmo and public datasets, and quantitative analysis was carried out. It was found that the proposed model achieved advanced recognition performance, and the ablation experiments verified the importance of different components of the model. The proposed model and dataset are beneficial to the realization of human-vehicle affective interaction in intelligent cockpits in the future toward a better human experience.

Local and Global Spectral Features for Hyperspectral Image Classification

Hyperspectral images (HSI) contain powerful spectral characterization capabilities and are widely used especially for classification applications. However, the rich spectrum contained in HSI also increases the difficulty of extracting useful information, which makes the feature extraction method significant as it enables effective expression and utilization of the spectrum. Traditional HSI feature extraction methods design spectral features manually, which is likely to be limited by the complex spectral information within HSI. Recently, data-driven methods, especially the use of convolutional neural networks (CNNs), have shown great improvements in performance when processing image data owing to their powerful automatic feature learning and extraction abilities and are also widely used for HSI feature extraction and classification. The CNN extracts features based on the convolution operation. Nevertheless, the local perception of the convolution operation makes CNN focus on the local spectral features (LSF) and weakens the description of features between long-distance spectral ranges, which will be referred to as global spectral features (GSF) in this study. LSF and GSF describe the spectral features from two different perspectives and are both essential for determining the spectrum. Thus, in this study, a local-global spectral feature (LGSF) extraction and optimization method is proposed to jointly consider the LSF and GSF for HSI classification. To increase the relationship between spectra and the possibility to obtain features with more forms, we first transformed the 1D spectral vector into a 2D spectral image. Based on the spectral image, the local spectral feature extraction module (LSFEM) and the global spectral feature extraction module (GSFEM) are proposed to automatically extract the LGSF. The loss function for spectral feature optimization is proposed to optimize the LGSF and obtain improved class separability inspired by contrastive learning. We further enhanced the LGSF by introducing spatial relation and designed a CNN constructed using dilated convolution for classification. The proposed method was evaluated on four widely used HSI datasets, and the results highlighted its comprehensive utilization of spectral information as well as its effectiveness in HSI classification.

EEG emotion recognition using attention-based convolutional transformer neural network

EEG-based emotion recognition has become an important task in affective computing and intelligent interaction. However, how to effectively combine the spatial, spectral, and temporal distinguishable information of EEG signals to achieve better emotion recognition performance is still a challenge. In this paper, we propose a novel attention-based convolutional transformer neural network (ACTNN), which effectively integrates the crucial spatial, spectral, and temporal information of EEG signals, and cascades convolutional neural network and transformer in a new way for emotion recognition task. We first organized EEG signals into spatial–spectral–temporal representations. To enhance the distinguishability of features, spatial and spectral attention masks are learned for the representation of each time slice. Then, a convolutional module is used to extract local spatial and spectral features. Finally, we concatenate the features of all time slices, and feed them into the transformer-based temporal encoding layer to use multi-head self-attention for global feature awareness. The average recognition accuracy of the proposed ACTNN on two public datasets, namely SEED and SEED-IV, is 98.47% and 91.90% respectively, outperforming the state-of-the-art methods. Besides, to explore the underlying reasoning process of the model and its neuroscience relevance with emotion, we further visualize spatial and spectral attention masks. The attention weight distribution shows that the activities of prefrontal lobe and lateral temporal lobe of the brain, and the gamma band of EEG signals might be more related to human emotion. The proposed ACTNN can be employed as a promising framework for EEG emotion recognition.

PCA-Domain Fused Singular Spectral Analysis for Fast and Noise-Robust Spectral–Spatial Feature Mining in Hyperspectral Classification

The principal component analysis (PCA) and 2-D singular spectral analysis (2DSSA) are widely used for spectral domain and spatial domain feature extraction in hyperspectral images (HSI). However, PCA itself suffers from low efficacy if no spatial information is combined, whilst 2DSSA can extract the spatial information yet has a high computing complexity. As a result, we propose in this paper a PCA domain 2DSSA approach for spectral-spatial feature mining in HSI. Specifically, PCA and its variation, folded-PCA are utilized to fuse with the 2DSSA, as folded-PCA can extract both global and local spectral features. By applying 2DSSA only on a small number of PCA components, the overall computational complexity has been significantly reduced whilst preserving the discrimination ability of the features. In addition, with the effective fusion of spectral and spatial features, the proposed approach can work well on the uncorrected dataset without removing the noisy and water absorption bands, even under a small number of training samples. Experiments on two publicly available datasets have fully demonstrated the superiority of the proposed approach, in comparison to several state-of-the-art HSI classification methods and deep-learning models.

A graph-based approach for simultaneous semantic and instance segmentation of plant 3D point clouds.

Accurate simultaneous semantic and instance segmentation of a plant 3D point cloud is critical for automatic plant phenotyping. Classically, each organ of the plant is detected based on the local geometry of the point cloud, but the consistency of the global structure of the plant is rarely assessed. We propose a two-level, graph-based approach for the automatic, fast and accurate segmentation of a plant into each of its organs with structural guarantees. We compute local geometric and spectral features on a neighbourhood graph of the points to distinguish between linear organs (main stem, branches, petioles) and two-dimensional ones (leaf blades) and even 3-dimensional ones (apices). Then a quotient graph connecting each detected macroscopic organ to its neighbors is used both to refine the labelling of the organs and to check the overall consistency of the segmentation. A refinement loop allows to correct segmentation defects. The method is assessed on both synthetic and real 3D point-cloud data sets of Chenopodium album (wild spinach) and Solanum lycopersicum (tomato plant).

Multi-Scale LBP Texture Feature Learning Network for Remote Sensing Interpretation of Land Desertification

Land desertification is a major challenge to global sustainable development. Therefore, the timely and accurate monitoring of the land desertification status can provide scientific decision support for desertification control. The existing automatic interpretation methods are affected by factors such as “same spectrum different matter”, “different spectrum same object”, staggered distribution of desertification areas, and wide ranges of ground objects. We propose an automatic interpretation method for the remote sensing of land desertification that incorporates multi-scale local binary pattern (MSLBP) and spectral features based on the above issues. First, a multi-scale convolutional LBP feature extraction network is designed to obtain the spatial texture features of remote sensing images and fuse them with spectral features to enhance the feature representation capability of the model. Then, considering the continuity of the distribution of the same kind of ground objects in local space, we designed an adaptive median filtering method to process the probability map of the extreme learning machine (ELM) classifier output to improve the classification accuracy. Four typical datasets were developed using GF-1 multispectral imagery with the Horqin Left Wing Rear Banner as the study area. Experimental results on four datasets show that the proposed method solves the problem of ill classification and omission in classifying the remote sensing images of desertification, effectively suppresses the effects of “homospectrum” and “heterospectrum”, and significantly improves the accuracy of the remote sensing interpretation of land desertification.