Hyperspectral Image Classification Research Articles

MDvT: introducing mobile three-dimensional convolution to a vision transformer for hyperspectral image classification

ABSTRACT Hyperspectral images carry numerous spectral bands, and their wealth of band data is a valuable source of information for the accurate classification of ground objects. Three-dimensional (3D) convolution, although an excellent spectral information extraction method, is limited by its huge number of parameters and long model training time. To allow better integration of 3D convolution with the most popular transformer models currently available, a new architecture called mobile 3D convolutional vision transformer (MDvT) is proposed. The MDvT introduces inverted residual structure to reduce the number of model parameters and balance the data mining efficiency of low-dimensional data input. Simultaneously, a square patch is used to cut the sequence of tokens to accelerate the model operation. Through extensive experiments, we evaluated the classification overall performance of the proposed MDvT on the WHU-Hi and Pavia University datasets, and demonstrated significant improvements in classification accuracy and model runtime compared with classical deep learning models. It is worth noting that compared with directly integrating 3D convolution into the transformer model, the MDvT architecture improves the accuracy while reducing the time to train an epoch by approximately 58.54%. To facilitate the reproduction of the work in this paper, the model code is available at https://github.com/gloryofroad/MDvT.

Attention-Embedded Triple-Fusion Branch CNN for Hyperspectral Image Classification

Hyperspectral imaging (HSI) is widely used in various fields owing to its rich spectral information. Nonetheless, the high dimensionality of HSI and the limited number of labeled data remain significant obstacles to HSI classification technology. To alleviate the above problems, we propose an attention-embedded triple-branch fusion convolutional neural network (AETF-Net) for an HSI classification. The network consists of a spectral attention branch, a spatial attention branch, and a multi-attention fusion branch (MAFB). The spectral branch introduces the cross-channel attention to alleviate the band redundancy problem in high dimensions, while the spatial branch preserves the location information of features and eliminates interfering image elements by a bi-directional spatial attention module. These pre-extracted spectral and spatial attention features are then embedded into a novel MAFB with large kernel decomposition technique. The proposed AETF-Net achieves multi-attention features reuse and extracts more representative and discriminative features. Experimental results on three well-known datasets demonstrate the superiority of the method AETF-Net.

Discrimination between normal and necrotic small intestinal tissue using hyperspectral imaging and unsupervised classification.

Objective and automatic clinical discrimination of normal and necrotic sites of small intestinal tissue remains challenging. In this study, hyperspectral imaging (HSI) and unsupervised classification techniques were used to distinguish normal and necrotic sites of small intestinal tissues. Small intestinal tissue hyperspectral images of eight Japanese large-eared white rabbits were acquired using a visible near-infrared hyperspectral camera, and K-means and density peaks (DP) clustering algorithms were used to differentiate between normal and necrotic tissue. The three cases in this study showed that the average clustering purity of the DP clustering algorithm reached 92.07% when the two band combinations of 500-622 and 700-858 nm were selected. The results of this study suggest that HSI and DP clustering can assist physicians in distinguishing between normal and necrotic sites in the small intestine in vivo.

Quaternion convolutional neural networks for hyperspectral image classification

Quaternion convolutional neural networks (QCNNs) can capture quaternion features, which contain not only the contextual information among quaternion feature units but also utilize the quaternion algebra inside the quaternion feature units to express structural information. However, building efficient QCNNs for hyperspectral image (HSI) classification is a challenge due to the lack of methods to map real features into quaternion features and the missing key quaternion modules. This paper proposes methodologies to build QCNNs specifically for HSI classification and designs several key quaternion modules. Firstly, a novel quaternion feature encoder is presented to map real HSI features into quaternion features. Secondly, a paradigm is designed to conveniently modify classical convolutional neural networks to implement QCNNs for HSI classification. Thirdly, we propose a new separable quaternion convolutional neural network (SQNet) based on presented advanced quaternion modules, such as separable quaternion convolutions and quaternion attention mechanisms. In addition, the computational rules in quaternion neurons are improved to enhance the adaptability of QCNNs. Experiments conducted on four widely used HSI datasets indicate that the implemented QCNNs and the proposed SQNet achieve satisfactory results for HSI classification with limited training samples.

Application of XGBoost for Fast Identification of Typical Industrial Organic Waste Samples with Near-Infrared Hyperspectral Imaging

Waste material identification is an essential part of waste recycling and treatment. Hyperspectral imaging (HSI) enables fast, accurate, nondestructive, and non-invasive identification of waste materials. In this study, HSI-based classification of typical industrial organic waste that cannot be sorted via traditional methods has been explored, namely, leather, paper, plastic, rubber, textile, and wood. The extreme gradient boosting (XGBoost) algorithm, a supervised machine learning algorithm that has never been investigated for waste identification-related fields, was adopted. The results show that XGBoost obtained a higher pixel-wise weighted average F1-score of 82.72% and a faster prediction time of 270 ms for the tested images compared with the commonly used partial least squares-discriminant analysis (77.83% and 444 ms). XGBoost was more effective and efficient in aiding HSI identification and classification of industrial organic waste. The technique can be a significant advancement in the development of an online sorting or identification platform, affording significant labor cost reduction, time savings, and the provision of a stable, accurate, and rapid method for waste intelligent identification.

Ensemble of Support Vector Machines for spectral-spatial classification of hyperspectral and multispectral images

Ensemble of Support Vector Machines for spectral-spatial classification of hyperspectral and multispectral images

Multi-scale spectral-spatial dual-transformer network for hyperspectral image classification

ABSTRACT Deep learning methods have shown great advantages in hyperspectral image (HSI) classification tasks. In particular, convolutional neural network (CNN)-based methods for HSI classification have made great progress. However, it is difficult for CNN to process long-range spatial and spectral information. Recently, a novel deep learning model called transformer has demonstrated its potential to replace CNN in various classification tasks with its amazing performance. In this letter, a multi-scale spectral-spatial dual-transformer network (MS3DT) is proposed to deeply consider the spectral-spatial features via transformer. Specifically, MS3DT consists of a feature pyramid network (FPN), a spectral transformer subnetwork (SPECT) and a spatial transformer subnetwork (SPAT). To utilize the complementary multi-scale characteristics, we introduce FPN to capture shallow-to-deep and spectral-spatial features. To improve the representational capacity in spatial and spectral domains, SPECT is exploited to extract long-range spectral correlation over local spectral features, and SPAT is designed to explore the contextual information for better refinement. Therefore, MS3DT is able to adaptively recalibrate the nonlinear interdependence of shallow-to-deep spectral-spatial features by merging the two subnetworks. Experiments on two widely used HSI datasets show that the MS3DT can outperform the state-of-the-art (SOTA) algorithm. The source codes will be available at https://github.com/RsAI-lab/MSSSDT.

Max–min distance embedding for unsupervised hyperspectral image classification in the satellite Internet of Things system

In modern society, hyperspectral images have widely been used in many applications. In the earth observation system, hyperspectral images are acquired by hyperspectral sensors and transmitted to the cloud computing center for analysis, which are the part of satellite Internet of Things(IoT) to build a large scale land-cover surveillance system. Thus, during recent years, researchers throughout academia and industry have been advancing the classification of hyperspectral images. Luckily for swamped remote sensing researchers, new technology is allowing brands to extract useful knowledge using artificial intelligence to enhance supervised hyperspectral image classification technology. However, conventional methods that label and process hyperspectral images manually are labor and time-consuming. Hence, in this paper, we proposed an unsupervised hyperspectral image classification framework. The paper differs from the existing methods by three aspects: (1) We adopt multi-scale spatial features to enhance the discriminability of data. (2) We propose deep autoencoder with max–min distance embedding to extract features and reduce dimensions of data, which is more suitable for large-scale data clustering. (3) We apply k-means clustering method to get original classification, then optimize the results by the guided filter. Experimental results on two widely used hyperspectral data, Salinas and Pavia University, show the competitive performance obtained by the proposed framework compared with other approaches.

Broad Graph Convolutional Neural Network and Its Application in Hyperspectral Image Classification

A fast and effective machine learning method, the broad learning system (BLS), has been successfully used for hyperspectral image (HSI) classification with good results. However, the original BLS cannot fully utilize the spatial information of HSI, and the linear sparse features of mapping nodes (MFs) have insufficient ability to characterize HSI. Thus, a broad graph convolutional neural network (BGCNN) is proposed for solving the aforementioned issues. In the BGCNN, the graph convolution operation is first used to capture the nonlinear spectral-spatial features, instead of only the linear sparse autoencoder in BLS. Then, the spectral-spatial features are expanded with a graph convolution operation, which further enhances the feature representation capability. Finally, the ridge regression theory is exploited to acquire the output weights. Experiments on four real HSI datasets show that our proposed BGCNN outperforms several state-of-the-art classification methods on the classification accuracy with a relatively less consumed time.

Lightweight Parallel Octave Convolutional Neural Network for Hyperspectral Image Classification

Although most deep learning-based methods have achieved excellent performance for hyperspectral image (HSI) classification, they are often limited by complex networks and require massive training samples in practical applications. Therefore, designing an efficient, lightweight model to obtain better classification results under small samples situations remains a challenging task. To alleviate this problem, a novel, lightweight parallel octave convolutional neural network (LPOCNN) for HSI classification is proposed in this paper. First, the HSI data is preprocessed to construct two three-dimensional (3D) patch cubes with different spatial and spectral scales for each central pixel, removing redundancy and focusing on extracting spatial features and spectral features, respectively. Next, two non- deep parallel branches are created for the two inputs, which design octave convolution rather than classical 3D convolution to facilitate light weighting of the model. Then two-dimensional convolutional neural network is used to extract deeper spectral-spatial features when fusing spectral-spatial features from different parallel layers. Moreover, the spectral-spatial attention is designed to promote the classification performance even further by adaptively adjusting the weights of different spectral-spatial features according to their contribution to classification. Experiments show that our suggested LPOCNN acquires a significant advantage on classification performance over other competitive methods under small sample situations.

Exploring the Relationship Between Center and Neighborhoods: Central Vector Oriented Self-Similarity Network for Hyperspectral Image Classification

To mine the spectral-spatial information of target pixel in hyperspectral image classification (HSIC), convolutional neural network (CNN)-based models widely adopt patch-based input pattern, where a patch represents its central pixel and the neighbor pixels play auxiliary roles in the classification process. However, compared to the central pixel, its neighbor pixels often have different contributions for classification. Although many existing patch-based CNNs could adaptively emphasize the spatial neighbor information, most of them ignore the latent relationship between the center pixel and its neighbor pixels. Moreover, efficient spectral-spatial feature extraction has been a difficult yet vital topic for HSIC. To address the mentioned problems, a central vector oriented self-similarity network (CVSSN) is proposed for HSIC. Specifically, based on two similarity measures, we firstly design an adaptive weight addition based spectral vector self-similarity module (AWA-SVSS) in input space and a Euclidean distance based feature vector self-similarity module (ED-FVSS) in feature space to fully mine the central vector oriented spatial relationships. Besides, a spectral-spatial information fusion module (SSIF) is formulated as a new pattern to fuse the central 1D spectral vector and the corresponding 3D patch for efficient spectral-spatial feature learning of the subsequent modules. Moreover, we implement a channel spatial separation convolution module (CSS-Conv) and a scale information complementary convolution module (SIC-Conv) for efficient spectral-spatial feature learning. Extensive experimental results on four popular HSI data sets demonstrate the effectiveness and efficiency of the proposed method compared with other state-of-the-art methods. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lms-07/CVSSN</uri>

DGSSC: A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspectral Imagery

In recent years, hyperspectral image classification (HSIC) has achieved impressive progress with emerging studies on deep learning models. However, the classification performance downgrades due to the limited number of annotated samples, especially for minority classes. Notably, the imbalanced data dilemma is familiar in remote sensing hyperspectral image because the ground objects are commonly distributed without evenness. Therefore, this paper proposes a novel deep generative spectral-spatial classifier (DGSSC) for addressing the issues of imbalanced HSIC. Specifically, the DGSSC comprises three components, a two-stage encoder, a decoder, and a classifier, which are trained in an end-to-end manner. In particular, to exploit the abundant spectral-spatial features with relatively low computational complexity, the first stage of the encoder comprises successive three-dimensional (3D) and two-dimensional (2D) convolutions, exploring the spectral-spatial and deep spatial information. In addition, the second stage involves the deep latent variable model to achieve minority-class data augmentation. Furthermore, a patch distance-based reconstruction loss function is designed to facilitate the outputs of the decoder being more similar to the input 3D patch samples. The proposed DGSSC can outperform the state-of-the-art methods on three benchmark datasets, especially with its more robust prediction results. For instance, the DGSSC achieves a remarkable 97.85% mean overall accuracy with 0.24% standard deviation over ten independent runs with randomly selected imbalanced 1% training samples on the University of Pavia dataset.

Masked Graph Convolutional Network for Small Sample Classification of Hyperspectral Images

The deep learning method has achieved great success in hyperspectral image classification, but the lack of labeled training samples still restricts the development and application of deep learning methods. In order to deal with the problem of small samples in hyperspectral image classification, a novel small sample classification method based on rotation-invariant uniform local binary pattern (RULBP) features and a graph-based masked autoencoder is proposed in this paper. Firstly, the RULBP features of hyperspectral images are extracted, and then the k-nearest neighbor method is utilized to construct the graph. Furthermore, self-supervised learning is conducted on the constructed graph so that the model can learn to extract features more suitable for small sample classification. Since the self-supervised training mainly adopts the masked autoencoder method, only unlabeled samples are needed to complete the training. After training, only a small number of samples are used to fine-tune the graph convolutional network, so as to complete the classification of all nodes in the graph. A large number of classification experiments on three commonly used hyperspectral image datasets show that the proposed method could achieve higher classification accuracy with fewer labeled samples.

Hyperspectral images classification for white blood cells with attention-aided convolutional neural networks and fusion network

The classification of White blood cells (WBCs) plays an important role. However, the traditional method of blood smear analysis is laborious. This paper presented a classification method of WBCs based on hyperspectral images and Deep learning (DL). The U-net network was proposed to extract spectral features of WBCs region of interest (ROI) under the pseudo-color images with specific hyperspectral images (420.8, 557.2 and 667.4 nm). For spectral features and the pseudo-colour images of hyperspectral data, attention-aided one and two-dimensional convolutional neural networks were applied to establish WBCs classification models, respectively. The overall average accuracy can reach 94.20% and 92.60%, respectively. A fusion network was constructed to make full use of the spectral and image spatial features, and its classification accuracy reached 96.20%. In terms of overall average accuracy, the fusion network model was the optimal, but for individual types of WBCs, each network had its own unique advantages.

A multi-scale residual capsule network for hyperspectral image classification with small training samples

A multi-scale residual capsule network for hyperspectral image classification with small training samples

Local and Global Spectral Features for Hyperspectral Image Classification

Hyperspectral images (HSI) contain powerful spectral characterization capabilities and are widely used especially for classification applications. However, the rich spectrum contained in HSI also increases the difficulty of extracting useful information, which makes the feature extraction method significant as it enables effective expression and utilization of the spectrum. Traditional HSI feature extraction methods design spectral features manually, which is likely to be limited by the complex spectral information within HSI. Recently, data-driven methods, especially the use of convolutional neural networks (CNNs), have shown great improvements in performance when processing image data owing to their powerful automatic feature learning and extraction abilities and are also widely used for HSI feature extraction and classification. The CNN extracts features based on the convolution operation. Nevertheless, the local perception of the convolution operation makes CNN focus on the local spectral features (LSF) and weakens the description of features between long-distance spectral ranges, which will be referred to as global spectral features (GSF) in this study. LSF and GSF describe the spectral features from two different perspectives and are both essential for determining the spectrum. Thus, in this study, a local-global spectral feature (LGSF) extraction and optimization method is proposed to jointly consider the LSF and GSF for HSI classification. To increase the relationship between spectra and the possibility to obtain features with more forms, we first transformed the 1D spectral vector into a 2D spectral image. Based on the spectral image, the local spectral feature extraction module (LSFEM) and the global spectral feature extraction module (GSFEM) are proposed to automatically extract the LGSF. The loss function for spectral feature optimization is proposed to optimize the LGSF and obtain improved class separability inspired by contrastive learning. We further enhanced the LGSF by introducing spatial relation and designed a CNN constructed using dilated convolution for classification. The proposed method was evaluated on four widely used HSI datasets, and the results highlighted its comprehensive utilization of spectral information as well as its effectiveness in HSI classification.

Graph Convolutional Network Using Adaptive Neighborhood Laplacian Matrix for Hyperspectral Images with Application to Rice Seed Image Classification

Graph convolutional neural network architectures combine feature extraction and convolutional layers for hyperspectral image classification. An adaptive neighborhood aggregation method based on statistical variance integrating the spatial information along with the spectral signature of the pixels is proposed for improving graph convolutional network classification of hyperspectral images. The spatial-spectral information is integrated into the adjacency matrix and processed by a single-layer graph convolutional network. The algorithm employs an adaptive neighborhood selection criteria conditioned by the class it belongs to. Compared to fixed window-based feature extraction, this method proves effective in capturing the spectral and spatial features with variable pixel neighborhood sizes. The experimental results from the Indian Pines, Houston University, and Botswana Hyperion hyperspectral image datasets show that the proposed AN-GCN can significantly improve classification accuracy. For example, the overall accuracy for Houston University data increases from 81.71% (MiniGCN) to 97.88% (AN-GCN). Furthermore, the AN-GCN can classify hyperspectral images of rice seeds exposed to high day and night temperatures, proving its efficacy in discriminating the seeds under increased ambient temperature treatments.

Multi-Scale Spectral-Spatial Attention Network for Hyperspectral Image Classification Combining 2D Octave and 3D Convolutional Neural Networks

Traditional convolutional neural networks (CNNs) can be applied to obtain the spectral-spatial feature information from hyperspectral images (HSIs). However, they often introduce significant redundant spatial feature information. The octave convolution network is frequently utilized instead of traditional CNN to decrease spatial redundant information of the network and extend its receptive field. However, the 3D octave convolution-based approaches may introduce extensive parameters and complicate the network. To solve these issues, we propose a new HSI classification approach with a multi-scale spectral-spatial network-based framework that combines 2D octave and 3D CNNs. Our method, called MOCNN, first utilizes 2D octave convolution and 3D DenseNet branch networks with various convolutional kernel sizes to obtain complex spatial contextual feature information and spectral characteristics, separately. Moreover, the channel and the spectral attention mechanisms are, respectively, applied to these two branch networks to emphasize significant feature regions and certain important spectral bands that comprise discriminative information for the categorization. Furthermore, a sample balancing strategy is applied to address the sample imbalance problem. Expansive experiments are undertaken on four HSI datasets, demonstrating that our MOCNN approach outperforms several other methods for HSI classification, especially in scenarios dominated by limited and imbalanced sample data.

Exponential linear units-guided Depthwise separable convolution network with cross attention mechanism for hyperspectral image classification

Hyperspectral images (HSI) are more informative than other remote sensing techniques and hence frequently utilized in numerous domains. Convolutional neural network algorithm provides outstanding performance in image processing and currently has established as the main approach in the field of HSI classification. HSI contain numerous channels with various spatial and spectral feature information, and simultaneously contain massive redundant information, leading to dimensional explosion or gradient disappearance in the process of classification. A multilayer network model based on Exponential Linear Units-guided Depthwise Separable Convolution is proposed in this paper to extract both spectral and spatial features from HSI at scales ranging, and the feature maps are then fed into a cross-attention mechanism for weight allocation to improve local feature information and optimize computational resource allocation. Extensive experiments on three well-known hyperspectral datasets are conducted, and the results show that the proposed network model can accurately complete the required HSI classification operation and outperforms other well-established techniques in terms of computing efficiency.

Nearest Neighboring Self-Supervised Learning for Hyperspectral Image Classification

Recently, state-of-the-art classification performance of natural images has been obtained by self-supervised learning (S2L) as it can generate latent features through learning between different views of the same images. However, the latent semantic information of similar images has hardly been exploited by these S2L-based methods. Consequently, to explore the potential of S2L between similar samples in hyperspectral image classification (HSIC), we propose the nearest neighboring self-supervised learning (N2SSL) method, by interacting between different augmentations of reliable nearest neighboring pairs (RN2Ps) of HSI samples in the framework of bootstrap your own latent (BYOL). Specifically, there are four main steps: pretraining of spectral spatial residual network (SSRN)-based BYOL, generation of nearest neighboring pairs (N2Ps), training of BYOL based on RN2P, final classification. Experimental results of three benchmark HSIs validated that S2L on similar samples can facilitate subsequent classification. Moreover, we found that BYOL trained on an un-related HSI can be fine-tuned for classification of other HSIs with less computational cost and higher accuracy than training from scratch. Beyond the methodology, we present a comprehensive review of HSI-related data augmentation (DA), which is meaningful to future research of S2L on HSIs.