Joint Spectral-spatial Feature Research Articles

Spatial-spectral feature extraction for in-field chlorophyll content estimation using hyperspectral imaging

In-situ leaf chlorophyll content (LCC) estimation based on hyperspectral imaging (HSI) is crucial to track the growth status of crops for field management. However, spatial and spectral features of HSI data, suffering from interference of growth dynamic effect and soil, pose the challenge on accuracy and robustness of LCC estimation in several years and growth stages. Therefore, a joint spectral-spatial feature extraction method was proposed by cascade of three-dimensional convolutional neural network (3DCNN) and long short-term memory (LSTM) to reduce the interference for optimising the LCC estimation. Firstly, crop pixels were separated from soil with vegetation index segmentation method. Secondly, when raw images and segmented pixels were input, sensitive bands were selected by random frog (RF bands), and 3DCNN-LSTM was used to extract the joint spectral-spatial features. Finally, models established by RF bands, 3DCNN and 3DCNN-LSTM were compared, and robustness in individual years and stages was validated. Results showed that RF bands and 3DCNN obtained RP2 of 0.76 and 0.84 when not segmented. After segmentation, performance of 3DCNN improved (RP2 = 0.85) compared to RF bands (RP2 = 0.80). Spectral-spatial features by 3DCNN reduced the interference of soil. 3DCNN-LSTM without and with segmentation obtained good performance with RP2 of 0.95 and 0.96, and the proposed method could reduce the image segmentation process. The optimal model achieved RP2 above 0.93 in individual years (RP2 = 0.96 in 2021, RP2 = 0.94 in 2021) and RP2 in the range of 0.87–0.97 at individual stages. This paper provides a method to track growth variability between soil and crop for the LCC estimation optimisation.

Deep hierarchical spectral-spatial feature fusion for hyperspectral image classification based on convolutional neural network

Joint spectral-spatial feature extraction has been proven to be the most effective part of hyperspectral image (HSI) classification. But, due to the mixing of informative and noisy bands in HSI, joint spectral-spatial feature extraction using convolutional neural network (CNN) may lead to information loss and high computational cost. More specifically, joint spectral-spatial feature extraction from excessive bands may cause loss of spectral information due to the involvement of convolution operation on non-informative spectral bands. Therefore, we propose a simple yet effective deep learning model, named deep hierarchical spectral-spatial feature fusion (DHSSFF), where spectral-spatial features are exploited separately to reduce the information loss and fuse the deep features to learn the semantic information. It makes use of abundant spectral bands and few informative bands of HSI for spectral and spatial feature extraction, respectively. The spectral and spatial features are extracted through 1D CNN and 3D CNN, respectively. To validate the effectiveness of our model, the experiments have been performed on five well-known HSI datasets. Experimental results demonstrate that the proposed method outperforms other state-of-the-art methods and achieved 99.17%, 98.84%, 98.70%, 99.18%, and 99.24% overall accuracy on Kennedy Space Center, Botswana, Indian Pines, University of Pavia, and Salinas datasets, respectively.

Effective hyperspectral image classification based on segmented PCA and 3D-2D CNN leveraging multibranch feature fusion

ABSTRACT We present an innovative hyperspectral image (HSI) classification method addressing challenges posed by closely spaced wavelength bands. Our approach combines 3D-2D convolutional neural networks (CNNs) with multi-branch feature fusion for improved spectral-spatial feature extraction. Using segmented principal component analysis (Seg-PCA), we reduce HSIs’ spectral dimensions into global and local intrinsic characteristics. The integration of 3D and 2D CNNs captures joint spectral-spatial features, while a multi-branch network extracts and merges diverse local features along the spectral dimension. Our method outperforms existing approaches, achieving remarkable accuracy of 99.27%, 100%, and 99.99% on Indian Pines, Salinas Scene, and University of Pavia datasets, respectively.

Two-Branch Generative Adversarial Network With Multiscale Connections for Hyperspectral Image Classification

Hyperspectral image (HSI) classification has always drawn great attention in the field of remote sensing. Various deep learning models are in the ascendant and gradually applied to HSI classification. Nevertheless, limited-labeled and class-imbalanced datasets largely make the classifier prone to overfitting. To address the above problem, this article proposes a two-branch generative adversarial network with multiscale connections (TBGAN), which includes two generators to produce the spectral and spatial samples, respectively. Thereinto, the spectral generator is imbued with the self-attention mechanism to maximumly capture the long-term dependencies across the spectral bands. And meanwhile, an elaborated discriminator with two branches is devised in TBGAN for extracting the joint spectral-spatial features. Besides, the multiscale connections are placed between the discriminator and two generators to alleviate the instability problems caused by the inherently backward propagation of gradients in GAN. Furthermore, a feature-matching term is added to the loss function to prevent the generators from overtraining upon the current discriminator, thereby further improving the stability of the network. Experiments upon three benchmark datasets demonstrate that TBGAN achieves an extremely competitive classification accuracy and exerts lower sensitivity to the training sample size compared with several state-of-the-art methods.

A Multibranch Crossover Feature Attention Network for Hyperspectral Image Classification

Recently, hyperspectral image (HSI) classification methods based on convolutional neural networks (CNN) have shown impressive performance. However, HSI classification still faces two challenging problems: the first challenge is that most existing classification approaches only focus on exploiting the fixed-scale convolutional kernels to extract spectral–spatial features, which leads to underutilization of information; the second challenge is that HSI contains a large amount of redundant information and noise, to a certain extent, which influences the classification performance of CNN. In order to tackle the above problems, this article proposes a multibranch crossover feature attention network (MCFANet) for HSI classification. The MCFANet involves two primary submodules: a cross feature extraction module (CFEM) and rearranged attention module (RAM). The former is devised to capture joint spectral–spatial features at different convolutional layers, scales and branches, which can increase the diversity and complementarity of spectral–spatial features, while the latter is constructed to spontaneously concentrate on recalibrating spatial-wise and spectral-wise feature responses, meanwhile exploit the shifted cascade operation to rearrange the obtained attention-enhanced features to dispel redundant information and noise, and thus, boost the classification performance. Compared with the state-of-the-art classification methods, massive experiments on four benchmark datasets demonstrate the meliority of our presented method.

The real-time and stack fusion enhanced dual-channel network with attention modules for fast hyperspectral image classification

Hyperspectral image (HSI) classification with convolutional neural network (CNN) exhibiting excellent performance, but the improvement of computationally efficient is yet a challenging research task. Therefore, two fusion strategies and two attention modules have been introduced to enhance for a dual-channel structure, which consists of 2-D convolution branch for pure spatial features and 3-D convolution branch for joint spectral-spatial features. The real-time fusion is applied to each convolution layer between two branches, which enhanced information interaction within the network. Meanwhile, the stacked fusion strategy with the adaptive weights is introduced to adjust the influence of the two branches on the final classification results. Furthermore, the spectral-spatial reconstruction attention module (SSRAM) and adaptive channel attention module (ACAM) are designed to enhance features representative capability for 3-D branch and 2-D branch, respectively. All approaches introduced above allow us to construct a dual-channel network consisting of small convolution kernels, which achieve HSI classification accurately and efficiently. Experimental results on three public HSI datasets show that the proposed network outperforms several state-of-the-art models in terms of classification accuracy and computational efficiency.

SpaSSA: Superpixelwise Adaptive SSA for Unsupervised Spatial-Spectral Feature Extraction in Hyperspectral Image.

Singular spectral analysis (SSA) has recently been successfully applied to feature extraction in hyperspectral image (HSI), including conventional (1-D) SSA in spectral domain and 2-D SSA in spatial domain. However, there are some drawbacks, such as sensitivity to the window size, high computational complexity under a large window, and failing to extract joint spectral-spatial features. To tackle these issues, in this article, we propose superpixelwise adaptive SSA (SpaSSA), that is superpixelwise adaptive SSA for exploiting local spatial information of HSI. The extraction of local (instead of global) features, particularly in HSI, can be more effective for characterizing the objects within an image. In SpaSSA, conventional SSA and 2-D SSA are combined and adaptively applied to each superpixel derived from an oversegmented HSI. According to the size of the derived superpixels, either SSA or 2-D singular spectrum analysis (2D-SSA) is adaptively applied for feature extraction, where the embedding window in 2D-SSA is also adaptive to the size of the superpixel. Experimental results on the three datasets have shown that the proposed SpaSSA outperforms both SSA and 2D-SSA in terms of classification accuracy and computational complexity. By combining SpaSSA with the principal component analysis (SpaSSA-PCA), the accuracy of land-cover analysis can be further improved, outperforming several state-of-the-art approaches.

Advances in Hyperspectral Image Classification Based on Convolutional Neural Networks: A Review

Hyperspectral image (HSI) classification has been one of the most important tasks in the remote sensing community over the last few decades. Due to the presence of highly correlated bands and limited training samples in HSI, discriminative feature extraction was challenging for traditional machine learning methods. Recently, deep learning based methods have been recognized as powerful feature extraction tool and have drawn a significant amount of attention in HSI classification. Among various deep learning models, convolutional neural networks (CNNs) have shown huge success and offered great potential to yield high performance in HSI classification. Motivated by this successful performance, this paper presents a systematic review of different CNN architectures for HSI classification and provides some future guidelines. To accomplish this, our study has taken a few important steps. First, we have focused on different CNN architectures, which are able to extract spectral, spatial, and joint spectral-spatial features. Then, many publications related to CNN based HSI classifications have been reviewed systematically. Further, a detailed comparative performance analysis has been presented between four CNN models namely 1D CNN, 2D CNN, 3D CNN, and feature fusion based CNN (FFCNN). Four benchmark HSI datasets have been used in our experiment for evaluating the performance. Finally, we concluded the paper with challenges on CNN based HSI classification and future guidelines that may help the researchers to work on HSI classification using CNN.

A Biologically Interpretable Two-Stage Deep Neural Network (BIT-DNN) for Vegetation Recognition From Hyperspectral Imagery

Spectral-spatial based deep learning models have recently proven to be effective in hyperspectral image (HSI) classification for various earth monitoring applications such as land cover classification and agricultural monitoring. However, due to the nature of "black-box" model representation, how to explain and interpret the learning process and the model decision, especially for vegetation classification, remains an open challenge. This study proposes a novel interpretable deep learning model -- a biologically interpretable two-stage deep neural network (BIT-DNN), by incorporating the prior-knowledge (i.e. biophysical and biochemical attributes and their hierarchical structures of target entities) based spectral-spatial feature transformation into the proposed framework, capable of achieving both high accuracy and interpretability on HSI based classification tasks. The proposed model introduces a two-stage feature learning process: in the first stage, an enhanced interpretable feature block extracts the low-level spectral features associated with the biophysical and biochemical attributes of target entities; and in the second stage, an interpretable capsule block extracts and encapsulates the high-level joint spectral-spatial features representing the hierarchical structure of biophysical and biochemical attributes of these target entities, which provides the model an improved performance on classification and intrinsic interpretability with reduced computational complexity. We have tested and evaluated the model using four real HSI datasets for four separate tasks (i.e. plant species classification, land cover classification, urban scene recognition, and crop disease recognition tasks). The proposed model has been compared with five state-of-the-art deep learning models.

Edge-Conditioned Feature Transform Network for Hyperspectral and Multispectral Image Fusion

Despite recent advances achieved by deep learning techniques in the fusion of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI), it remains a challenge to reconstruct the high-spatial-resolution HSI (HR-HSI) with more accurate spatial details and less spectral distortions, since the low-level structure information such as sharp edges tends to be weakened or lost as the network depth grows. To tackle this issue, we creatively propose an edge-conditioned feature transform network (EC-FTN) in this article, which is mainly composed of three parts, namely, feature extraction network (FEN), feature fusion and transformation network (FFTN), and image reconstruction network (IRN). First, two computationally efficient FENs with 3-D convolutions and reshaping layers are employed to extract the joint spectral-spatial features of input images. Then, the FFTN conditioned on the edge map prior can fuse and transform the features adaptively, in which a fusion node and several cascaded feature modulation modules (FMMs) equipped with feature-wise modulation layers are constructed. Specifically, the edge map is generated via transfer learning, i.e., by applying the Sobel operator to feature maps of the red-green-blue (RGB) version of HR-MSI resulting from the pretrained VGG16 model without extra training. Finally, the desired HR-HSI is recovered from the transformed features through IRN. Furthermore, we elaborately design a weighted combinatorial loss function consisting of mean absolute error, image gradient difference, and spectral angle terms to guide the training. Experiments on both ground-based and remotely sensed datasets demonstrate that our EC-FTN outperforms state-of-the-art methods in visual and quantitive evaluations, as well as in fine details reconstruction.

A combination method of stacked autoencoder and 3D deep residual network for hyperspectral image classification

In comparison with conventional machine learning algorithms, deep learning can effectively express the deep features of remote sensing images. Considering the rich spectral and spatial information contained in hyperspectral images (HSIs), a combination method was proposed for HSI classification based on stacked autoencoder (SAE) and 3D deep residual network (3DDRN). Specifically, a SAE neural network was first built to reduce the dimensions of original HSIs. A 3D convolutional neural network (3DCNN) was then designed and the residual network module was introduced to build a 3DDRN. The dimension-reduced 3D HSI cubes were input into the 3DDRN to extract identifiable joint spectral-spatial features. Finally, the deep features continuously identified by the 3DDRN were input to Softmax classification layer to realize the classification. In addition, Batch Normalization (BN) and Dropout were used during the learning process to avoid overfitting on training data. The training and test sets of Indian Pines (IP), Pavia University (PU) and Salinas (SA) hyperspectral data sets were selected as the modeling and verification data sources. Six classical classification algorithms were adopted for comparing our proposed method, specifically including conventional machine learning algorithms of Radial Basis Function-Support Vector Machine (RBF-SVM), Kernel Simultaneous Orthogonal Matching Pursuit (KSOMP) and Local Binary Pattern-K-Nearest Neighbor (LBP-KNN), and mainstream deep learning algorithms of Variational Autoencoder (VAE), Convolutional Neural Network (CNN) and Spectral-Spatial Residual Network (SSRN). The results showed that the overall accuracy (OA) reached 98.97%, 99.69% and 99.24%, respectively, only based on 10%, 5% and 1% of training samples for IP, PU and SA. Consequently, the proposed method shows a better classification performance, even in the case of limited samples.

A Spectral-Spatial Features Integrated Network for Hyperspectral Detection of Marine Oil Spill

Marine oil spills are one of the most serious problems of marine environmental pollution. Hyperspectral remote sensing has been proven to be an effective tool for monitoring marine oil spills. To make full use of spectral and spatial features, this study proposes a spectral-spatial features integrated network (SSFIN) and applies it for hyperspectral detection of a marine oil spill. Specifically, 1-D and 2-D convolutional neural network (CNN) models have been employed for the extraction of the spectral and spatial features, respectively. During the stage of spatial feature extraction, three consecutive convolution layers are concatenated to achieve the fusion of multilevel spatial features. Next, the extracted spectral and spatial features are concatenated and fed to the fully connected layer so as to obtain the joint spectral-spatial features. In addition, L2 regularization is applied to the convolution layer to prevent overfitting, and dropout operation is employed to the full connection layer to improve the network performance. The effectiveness of the method proposed here has firstly been verified on the Pavia University dataset with competitive classification experimental results. Eventually, the experimental results upon oil spill datasets demonstrate the strong capacity of oil spill detection by this method, which can effectively distinguish thick oil film, thin oil film, and seawater.

A Lightweight Spectral–Spatial Feature Extraction and Fusion Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification accuracy has been greatly improved by employing deep learning. The current research mainly focuses on how to build a deep network to improve the accuracy. However, these networks tend to be more complex and have more parameters, which makes the model difficult to train and easy to overfit. Therefore, we present a lightweight deep convolutional neural network (CNN) model called S2FEF-CNN. In this model, three S2FEF blocks are used for the joint spectral–spatial features extraction. Each S2FEF block uses 1D spectral convolution to extract spectral features and 2D spatial convolution to extract spatial features, respectively, and then fuses spectral and spatial features by multiplication. Instead of using the full connected layer, two pooling layers follow three blocks for dimension reduction, which further reduces the training parameters. We compared our method with some state-of-the-art HSI classification methods based on deep network on three commonly used hyperspectral datasets. The results show that our network can achieve a comparable classification accuracy with significantly reduced parameters compared to the above deep networks, which reflects its potential advantages in HSI classification.

A Multiscale Deep Learning Approach for High-Resolution Hyperspectral Image Classification

Hyperspectral imagery (HSI) has emerged as a highly successful sensing modality for a variety of applications ranging from urban mapping to environmental monitoring and precision agriculture. Despite the efforts made by the scientific community, developing reliable algorithms of HSI classification remains a challenging problem, especially for high-resolution HSI data where there is often larger intraclass variability combined with a scarcity of the ground-truth data and class imbalance. In recent years, deep neural networks have emerged as a promising strategy for problems of HSI classification where they have shown remarkable potential for learning joint spectral-spatial features efficiently via backpropagation. In this letter, we propose a deep learning strategy for HSI classification that combines different convolutional neural networks, especially designed to efficiently learn joint spatial-spectral features over multiple scales. Our method achieves an overall classification accuracy of 66.73% on the 2018 IEEE GRSS hyperspectral data set-a high-resolution data set that includes 20 urban land-cover and land-use classes.

An overview on spectral and spatial information fusion for hyperspectral image classification: Current trends and challenges

Hyperspectral images (HSIs) have a cube form containing spatial information in two dimensions and rich spectral information in the third one. The high volume of spectral bands allows discrimination between various materials with high details. Moreover, by utilizing the spatial features of image such as shape, texture and geometrical structures, the land cover discrimination will be improved. So, fusion of spectral and spatial information can significantly improve the HSI classification. In this work, the spectral-spatial information fusion methods are categorized into three main groups. The first group contains segmentation based methods where objects or super-pixels are used instead of pixels for classification or the obtained segmentation map is used for relaxation of the pixel-wise classification map. The second group consists of feature fusion methods which are divided into six sub-groups: features stacking, joint spectral-spatial feature extraction, kernel based classifiers, representation based classifiers, 3D spectral-spatial feature extraction and deep learning based classifiers. The third fusion methods are decision fusion based approaches where complementary information of several classifiers are contributed for achieving the final classification map. A review of different methods in each category, is presented. Moreover, the advantages and difficulties/disadvantages of each group are discussed. The performance of various fusion methods are assessed in terms of classification accuracy and running time using experiments on three popular hyperspectral images. The results show that the feature fusion methods although are time consuming but can provide superior classification accuracy compared to other methods. Study of this work can be very useful for all researchers interested in HSI feature extraction, fusion and classification.

Hyperspectral classification using deep fusion spectral–spatial features

The deep convolution neural network (CNN) has been of great interest in hyperspectral image classification recently. Current CNN-based approaches adopt a two- or three-dimensional convolution network to extract spectral–spatial features from local pixel neighborhoods. High computation cost and overfitting problems should be handled carefully for these large-scale networks. A compact one-dimensional (1-D) CNN-based method is proposed to explore the joint spectral–spatial features by employing the pixel coordinates as the spatial information. Furthermore, two kinds of CNN architecture for spectral–spatial feature fusion are compared. To be specific, the shallow fusion model (SF-CNN) concatenates the spatial information to the spectral features before they are fed into the final fully connected layer, whereas the deep fusion model (DF-CNN) combines the spectral and the spatial information in an early stage and extracts the high-level spectral-spatial features by the 1-D convolution layers. The experimental results demonstrate that the DF-CNN method provides competitive classification results to state-of-the-art methods. Moreover, attributed to the concise spatial information and the effective feature fusion structure, the proposed method is economical in terms of computation cost when compared with the current CNN-based spectral-spatial methods.

Active Transfer Learning Network: A Unified Deep Joint Spectral–Spatial Feature Learning Model for Hyperspectral Image Classification

Deep learning has recently attracted significant attention in the field of hyperspectral images (HSIs) classification. However, the construction of an efficient deep neural network (DNN) mostly relies on a large number of labeled samples being available. To address this problem, this paper proposes a unified deep network, combined with active transfer learning that can be well-trained for HSIs classification using only minimally labeled training data. More specifically, deep joint spectral-spatial feature is first extracted through hierarchical stacked sparse autoencoder (SSAE) networks. Active transfer learning is then exploited to transfer the pre-trained SSAE network and the limited training samples from the source domain to the target domain, where the SSAE network is subsequently fine-tuned using the limited labeled samples selected from both source and target domain by corresponding active learning strategies. The advantages of our proposed method are threefold: 1) the network can be effectively trained using only limited labeled samples with the help of novel active learning strategies; 2) the network is flexible and scalable enough to function across various transfer situations, including cross-dataset and intra-image; 3) the learned deep joint spectral-spatial feature representation is more generic and robust than many joint spectral-spatial feature representation. Extensive comparative evaluations demonstrate that our proposed method significantly outperforms many state-of-the-art approaches, including both traditional and deep network-based methods, on three popular datasets.

Divide-and-Conquer Dual-Architecture Convolutional Neural Network for Classification of Hyperspectral Images

Convolutional neural network (CNN) is well-known for its powerful capability on image classification. In hyperspectral images (HSIs), fixed-size spatial window is generally used as the input of CNN for pixel-wise classification. However, single fixed-size spatial architecture hinders the excellent performance of CNN due to the neglect of various land-cover distributions in HSIs. Moreover, insufficient samples in HSIs may cause the overfitting problem. To address these problems, a novel divide-and-conquer dual-architecture CNN (DDCNN) method is proposed for HSI classification. In DDCNN, a novel regional division strategy based on local and non-local decisions is devised to distinguish homogeneous and heterogeneous regions. Then, for homogeneous regions, a multi-scale CNN architecture with larger spatial window inputs is constructed to learn joint spectral-spatial features. For heterogeneous regions, a fine-grained CNN architecture with smaller spatial window inputs is constructed to learn hierarchical spectral features. Moreover, to alleviate the problem of insufficient training samples, unlabeled samples with high confidences are pre-labeled under adaptively spatial constraint. Experimental results on HSIs demonstrate that the proposed method provides encouraging classification performance, especially region uniformity and edge preservation with limited training samples.

Learning and Transferring Deep Joint Spectral–Spatial Features for Hyperspectral Classification

Feature extraction is of significance for hyperspectral image (HSI) classification. Compared with conventional hand-crafted feature extraction, deep learning can automatically learn features with discriminative information. However, two issues exist in applying deep learning to HSIs. One issue is how to jointly extract spectral features and spatial features, and the other one is how to train the deep model when training samples are scarce. In this paper, a deep convolutional neural network with two-branch architecture is proposed to extract the joint spectral–spatial features from HSIs. The two branches of the proposed network are devoted to features from the spectral domain as well as the spatial domain. The learned spectral features and spatial features are then concatenated and fed to fully connected layers to extract the joint spectral–spatial features for classification. When the training samples are limited, we investigate the transfer learning to improve the performance. Low and mid-layers of the network are pretrained and transferred from other data sources; only top layers are trained with limited training samples extracted from the target scene. Experiments on Airborne Visible/Infrared Imaging Spectrometer and Reflective Optics System Imaging Spectrometer data demonstrate that the learned deep joint spectral–spatial features are discriminative, and competitive classification results can be achieved when compared with state-of-the-art methods. The experiments also reveal that the transferred features boost the classification performance.

Longwave infrared hyperspectral image classification via an ensemble method

ABSTRACTLongwave infrared hyperspectral images (LWIR-HSIs) classification is challenging, due to the poor imaging quality and low signal-to-noise ratio. A popular viewpoint is that abundant spatial contextual information can significantly improve the classification accuracies. However, it is quite difficult to determine what degree of spatial information is the most useful. In this article, we develop a novel ensemble-based classification method, which is able to fully leverage joint spectral-spatial features in different degrees. The proposed method contains three primary steps. First, a powerful edge-preserving filtering (EPF) approach, rolling guidance filtering (RGF), is utilized to generate several groups of diverse samples as well as enhance the quality of the LWIR-HSI data. Each group corresponds to a certain degree of spatial information. Subsequently, a series of individual classifiers are learned based on all groups of training samples, and each classifier could provide a single classification result for all test samples. Finally, we propose a new ensemble strategy, multi-classifier -statistic (MKS), to evaluate the contributions of individual learners (ILs). The final classification results are obtained based on the outputs of RGF and MKS. Experiments on a challenging LWIR-HSI data set verify the effectiveness of the proposed method, compared with some state-of-the-art HSI classification methods.