Joint Spatial-spectral Feature Research Articles

Identification of Turtle-Shell Growth Year Using Hyperspectral Imaging Combined with an Enhanced Spatial-Spectral Attention 3DCNN and a Transformer.

Turtle shell (Chinemys reecesii) is a prized traditional Chinese dietary therapy, and the growth year of turtle shell has a significant impact on its quality attributes. In this study, a hyperspectral imaging (HSI) technique combined with a proposed deep learning (DL) network algorithm was investigated for the objective determination of the growth year of turtle shells. The acquisition of hyperspectral images was carried out in the near-infrared range (948.72-2512.97 nm) from samples spanning five different growth years. To fully exploit the spatial and spectral information while reducing redundancy in hyperspectral data simultaneously, three modules were developed. First, the spectral-spatial attention (SSA) module was developed to better protect the spectral correlation among spectral bands and capture fine-grained spatial information of hyperspectral images. Second, the 3D convolutional neural network (CNN), more suitable for the extracted 3D feature map, was employed to facilitate the joint spatial-spectral feature representation. Thirdly, to overcome the constraints of convolution kernels as well as better capture long-range correlation between spectral bands, the transformer encoder (TE) module was further designed. These modules were harmoniously orchestrated, driven by the need to effectively leverage both spatial and spectral information within hyperspectral data. They collectively enhance the model's capacity to extract joint spatial and spectral features to discern growth years accurately. Experimental studies demonstrated that the proposed model (named SSA-3DTE) achieved superior classification accuracy, with 98.94% on average for five-category classification, outperforming traditional machine learning methods using only spectral information and representative deep learning methods. Also, ablation experiments confirmed the effectiveness of each module to improve performance. The encouraging results of this study revealed the potentiality of HSI combined with the DL algorithm as an efficient and non-destructive method for the quality control of turtle shells.

A Review of Hyperspectral Image Classification Based on Joint Spatial-spectral Features

Hyperspectral image classification technology is a basic work in the application of hyperspectral images. In recent years, with the innovation and development of hyperspectral image classification technology, the method and performance of hyperspectral image classification based on joint spatial-spectral features have made breakthroughs, and have gradually become the focus of researchers. In order to further promote the development of the spatial-spectral feature union class method and improve the classification accuracy of hyperspectral images, this paper summarizes the commonly used spatial-spectral union algorithms. Firstly, the introduction briefly outlines the background and research status of this field. Some common problems in the process of hyperspectral image classification are listed. Finally, some current hyperspectral image classification methods based on joint spatial-spectral features are introduced. The main roles and existing problems of spatial-spectral joint features in the field of hyperspectral image classification are summarized in detail, and the future research directions are prospected.

Two-Branch Pure Transformer for Hyperspectral Image Classification

Owing to its capability of building long-range dependencies and global context connections, Transformer has been used for hyperspectral image (HSI) classification. However, most of the existing Transformer-based HSI spatial–spectral classification methods consist of a convolutional neural network (CNN) and Transformer, which are used to extract the local and global information, respectively. In this study, to fully explore the potential of Transformer, a pure Transformer is investigated for HSI classification. First, a spatial Transformer (Spa-TR) is designed for HSI spatial classification, which learns the spatial features locally and globally by adopting the window partition and shifted window schemes. Especially, the self-attention computations are limited within the local windows and cross-windows. Second, to fully use the abundant spectral information in HSIs, a two-branch pure Transformer (i.e., Spa-Spe-TR) is proposed, which includes a spectral Transformer (Spe-TR) and a Spa-TR. The spectral sequence features learned by Spe-TR and the spatial features generated by Spa-TR are effectively fused with a branch fusion strategy, which explicitly and automatically measures the importance between the joint spatial–spectral features and improves the discriminability of the joint features. Experimental results on the two widely used HSI datasets (i.e., Pavia and Indian Pines) demonstrate the efficacy of the proposed methods in comparison with other state-of-the-art approaches.

Spectral-Spatial Features Extraction of Hyperspectral Remote Sensing Oil Spill Imagery Based on Convolutional Neural Networks

Marine oil spills accidents has caused a large amount of crude oil to leak into the marine environment and threaten marine ecological environment. Hyperspectral remote sensing images (HRSI) technology can quickly and accurately identify oil film of different thickness on marine surface. In order to overcome the traditional spectrum analysis method and space extraction method of long time sampling, calculation, analysis and other shortcomings. On account of the advantages of the spectral and spatial information in the field of HRSI classification, a spectral-spatial features extraction (SSFE) method based convolutional neural networks (CNNs) was proposed to analyse oil spills. In this way, one and two dimensional models based on convolutional neural networks (1D-CNN,2D-CNN) have been introduced as the spectral and spatial features extractor. When extracting spatial features, double-two convolution layers are connected to increasing nonlinearity and reduce the number of parameters. Furthermore, in order to address overfitting and imbalance samples, L2 regularization, class_weight and drouput is added to classes data modeling. More importantly, principal component analysis (PCA) is applied to data dimension reduction, 1D-CNN and 2D-CNN is combined into a unified model further extract the joint spatial-spectral features. To evaluate the effectiveness of the proposed approach, three hyperspectral datasets were utilized, including: University of Pavia dataset, oil spill area 1, oil spill area 2. Experimental results reveal that the proposed method have a very satisfactory performance and better distinguish oil spills.

Integrating Zhuhai-1 Hyperspectral Imagery With Sentinel-2 Multispectral Imagery to Improve High-Resolution Impervious Surface Area Mapping

Mapping impervious surface area (ISA) in an accurate and timely manner is essential for a variety of fields and applications, such as urban heat islands, hydrology, waterlogging, and urban planning and management. However, the large and complex urban landscapes pose great challenges in retrieving ISA information. Spaceborne hyperspectral (HS) remote sensing imagery provides rich spectral information with short revisit cycles, making it an ideal data source for ISA extraction from complex urban scenes. Nevertheless, insufficient single-band energy, the involvement of modulation transfer function (MTF), and the low signal-to-noise ratio (SNR) of spaceborne HS imagery usually result in poor image clarity and noises, leading to inaccurate ISA extraction. To address this challenge, we propose a new deep feature fusion-based classification method to improve 10 m resolution ISA mapping by integrating Zhuhai-1 HS imagery with Sentinel-2 multispectral (MS) imagery. We extract deep features that include spectral and spatial features, respectively, from MS and HS imagery via a 2-D convolutional neural network (CNN), aiming to increase feature diversity and improve the model’s recognition capability. The Sentinel-2 imagery is used to enhance the spatial information of the Zhuhai-1 HS image, improving the urban ISA retrieval by reducing the impact of noises. By combining the deep spatial features and deep spectral features, we obtain joint spatial-spectral features, leading to high classification accuracy and robustness. We test the proposed method in two highly urbanized study areas that cover Foshan city and Wuhan city, China. The results reveal that the proposed method obtains an overall accuracy of 96.72% and 96.75% in the two study areas, 18.78% and 8.66% higher than classification results with only HS imagery as input. The final ISA extraction overall accuracy is 95.42% and 95.50% in the two study areas, the highest among the comparison methods.

Aggregated-Attention Transformation Network for Hyperspectral Image Classification

Convolutional neural networks (CNNs) have achieved great success in hyperspectral image (HSI) classification. However, most CNNs require a substantial amount of manually labeled training data to yield good performances, and the expert annotation of HSI data is labor-intensive, but existing CNNs fail to effectively extract the joint spatial-spectral features from a small training set. To tackle this issue, inspired by the attention mechanism, an aggregated-attention transformation network (AATN) was proposed for HSI classification. This network takes advantage of the split-transform-merge strategy to change the number of the cardinality (the size of transformation set) for improving generalization ability with a small number of training samples while alleviating the overfitting phenomenon. Moreover, each aggregation attention module is designed to explore the attention across spatial-spectral features of different receptive fields, and interactively guide the recalibration of discriminative features. By retaining the uniform architecture, AATN can be used for HSI classification tasks without introducing excessive computational costs, and the proposed framework can learn representations with limited training samples due to the effectiveness and practicality of the attention-driven features. The proposed AATN obtains competitive performance compared with some state-of-the-art methods in qualitative and quantitative experiments on four benchmark HSI datasets.

Joint and Progressive Subspace Analysis (JPSA) With Spatial-Spectral Manifold Alignment for Semisupervised Hyperspectral Dimensionality Reduction.

Conventional nonlinear subspace learning techniques (e.g., manifold learning) usually introduce some drawbacks in explainability (explicit mapping) and cost effectiveness (linearization), generalization capability (out-of-sample), and representability (spatial-spectral discrimination). To overcome these shortcomings, a novel linearized subspace analysis technique with spatial-spectral manifold alignment is developed for a semisupervised hyperspectral dimensionality reduction (HDR), called joint and progressive subspace analysis (JPSA). The JPSA learns a high-level, semantically meaningful, joint spatial-spectral feature representation from hyperspectral (HS) data by: 1) jointly learning latent subspaces and a linear classifier to find an effective projection direction favorable for classification; 2) progressively searching several intermediate states of subspaces to approach an optimal mapping from the original space to a potential more discriminative subspace; and 3) spatially and spectrally aligning a manifold structure in each learned latent subspace in order to preserve the same or similar topological property between the compressed data and the original data. A simple but effective classifier, that is, nearest neighbor (NN), is explored as a potential application for validating the algorithm performance of different HDR approaches. Extensive experiments are conducted to demonstrate the superiority and effectiveness of the proposed JPSA on two widely used HS datasets: 1) Indian Pines (92.98%) and 2) the University of Houston (86.09%) in comparison with previous state-of-the-art HDR methods. The demo of this basic work (i.e., ECCV2018) is openly available at https://github.com/danfenghong/ECCV2018_J-Play.

Atom-substituted tensor dictionary learning enhanced convolutional neural network for hyperspectral image classification

A novel sparse tensor dictionary learning algorithm and a convolutional neural network (CNN) classification method based on this algorithm are proposed for hyperspectral image (HSI) in this paper. The HSI is intuitively represented as a three-dimensional (3-D) cube, and utilizing the joint spatial-spectral information can significantly improve the accuracy of HSI classification. Therefore, the proposed atom-substituted tensor dictionary learning (ASTDL) algorithm utilizes tensor techniques to extract 3-D joint spatial-spectral features from HSI cubes directly. Sparsity constraint is enforced on the coefficient tensors, which obeys the sparsity attribute of HSI. The proposed ASTDL enhanced CNN (ASTDL-CNN) classification method utilizes a two-dimensional (2-D) CNN to extract deep features from the feature tensors obtained by the ASTDL, and to perform the pixel-wise classification. The use of ASTDL, which extracts intrinsic tensor features before CNN, alleviates within-class spatial-spectral variation and reduces the requirement of CNN for the labeled data. Whereas, the CNN works as a 3-D classifier, and provides the final classification results. Besides, we perform the majority vote on the classification map obtained by CNN to refine the classification. The performance of the proposed method is evaluated on three real HSI data sets. The competitive results to the compared state-of-the-art methods demonstrate that the ASTDL-CNN can provide accurate and robust classification results.

Recurrent Feedback Convolutional Neural Network for Hyperspectral Image Classification

Deep neural networks have achieved promising performance for hyperspectral image (HSI) classification. However, due to the limitation of the available labeled samples, the traditional deeper and wider neural networks usually cause the overfitting problem and lose the detailed information. To solve this problem, a brain-like structure, namely spatial attention-driven recurrent feedback convolutional neural network (SARFNN), is proposed by utilizing the recurrent feedback and attention mechanism structures, from which two deep models are further developed for HSI classification. First, a 2-D SARFNN (SARF2DNN) model is developed to learn the spatial features from HSI data. After that, to better exploit the 3-D characteristic, the 3-D version is extended from SARF2DNN, thus constructing an SARF3DNN model to extract joint spatial-spectral features. Moreover, with the help of the idea of brain-likeness, the recurrent feedback module is designed to recover information loss caused by deeper structure and the dimension reduction operation. The experimental results conducted on two HSI data sets show that our SARFNN architecture can achieve more competitive performance than other state-of-the-art algorithms.

Joint spectral-spatial hyperspectral classification based on transfer learning (SSTL) from red-green-blue (RGB) images

ABSTRACT Deep learning (DL) approaches are demonstrated excellent performance on hyperspectral images (HSIs) classification tasks. Nevertheless, the imbalance between the scant number of available training samples, and the data-driven requirements of DL becomes the major limitation. In this work, we propose a deep convolutional neural network (CNN) with two-branch architecture based on multiple transfer learning (TL) strategies. Specifically, we designate one-dimensional (1D) CNN and two-dimensional (2D) CNN to extract the spectral and spatial features of the HSIs, respectively. The feature vectors from two branches are then concatenated and fed to fully connected layers to compose the collaborative spectral-spatial features. We implement the classification tasks via training a soft-max layer on the joint spatial-spectral features. To accelerate the process of training models and save calculation consumption, we apply transfer learning (TL) strategies to the spatial branch. The innovation lies in transferring VGG-16 (Visual Geometry Group Network) model pretrained on RGB (Red-Green-Blue) images datasets (ImageNet) to HSIs target domain. To investigate the effectiveness of the transfer learning between heterogeneous physical imaging camera, we design diverse regulating transferring strategies to explore the best performance. The classification results testing on the public HSIs datasets demonstrate our joint spectral-spatial based on transfer learning (SSTL) model performs excellently compared with other state-of-the-art methods.

Early Detection of Plant Viral Disease Using Hyperspectral Imaging and Deep Learning.

Early detection of grapevine viral diseases is critical for early interventions in order to prevent the disease from spreading to the entire vineyard. Hyperspectral remote sensing can potentially detect and quantify viral diseases in a nondestructive manner. This study utilized hyperspectral imagery at the plant level to identify and classify grapevines inoculated with the newly discovered DNA virus grapevine vein-clearing virus (GVCV) at the early asymptomatic stages. An experiment was set up at a test site at South Farm Research Center, Columbia, MO, USA (38.92 N, −92.28 W), with two grapevine groups, namely healthy and GVCV-infected, while other conditions were controlled. Images of each vine were captured by a SPECIM IQ 400–1000 nm hyperspectral sensor (Oulu, Finland). Hyperspectral images were calibrated and preprocessed to retain only grapevine pixels. A statistical approach was employed to discriminate two reflectance spectra patterns between healthy and GVCV vines. Disease-centric vegetation indices (VIs) were established and explored in terms of their importance to the classification power. Pixel-wise (spectral features) classification was performed in parallel with image-wise (joint spatial–spectral features) classification within a framework involving deep learning architectures and traditional machine learning. The results showed that: (1) the discriminative wavelength regions included the 900–940 nm range in the near-infrared (NIR) region in vines 30 days after sowing (DAS) and the entire visual (VIS) region of 400–700 nm in vines 90 DAS; (2) the normalized pheophytization index (NPQI), fluorescence ratio index 1 (FRI1), plant senescence reflectance index (PSRI), anthocyanin index (AntGitelson), and water stress and canopy temperature (WSCT) measures were the most discriminative indices; (3) the support vector machine (SVM) was effective in VI-wise classification with smaller feature spaces, while the RF classifier performed better in pixel-wise and image-wise classification with larger feature spaces; and (4) the automated 3D convolutional neural network (3D-CNN) feature extractor provided promising results over the 2D convolutional neural network (2D-CNN) in learning features from hyperspectral data cubes with a limited number of samples.

Attention Multibranch Convolutional Neural Network for Hyperspectral Image Classification Based on Adaptive Region Search

Convolutional neural networks (CNNs) have demonstrated outstanding performance on image classification. To classify the hyperspectral images (HSIs), existing CNN-based approaches commonly adopt the architecture using single or several fixed spatial windows as inputs. This kind of architecture may lose contextual information or incorporate heterogeneous information due to the neglect of various land-cover distributions in HSIs. To deal with this problem, a novel attention multibranch CNN method based on adaptive region search (RS-AMCNN) is proposed for HSI classification. In RS-AMCNN, sizes and locations of spatial windows are searched in the nonlocal candidate region adaptively according to sample-specific distribution. These flexible spatial windows are input into several branches of RS-AMCNN. In each branch, convolutional long short-term memories (ConvLSTMs) are merged into CNN from shallow to deep layers, which not only extracts joint spatial-spectral features, but also exploits complementary information among different layers. Then, a branch attention mechanism is devised to emphasize more discriminative branches and suppress less useful ones. It forces RS-AMCNN to extract multiscale and multicontextual attention features for classification. Finally, RS-AMCNN is optimized end-to-end by combining the losses from the ramose classifiers of different branches and the main classifier. Experiments carried on several benchmark HSI data sets demonstrate that RS-AMCNN provides promising classification performance, especially in edge preservation and region uniformity.

Lightweight Spectral–Spatial Squeeze-and- Excitation Residual Bag-of-Features Learning for Hyperspectral Classification

Of late, convolutional neural networks (CNNs) find great attention in hyperspectral image (HSI) classification since deep CNNs exhibit commendable performance for computer vision-related areas. CNNs have already proved to be very effective feature extractors, especially for the classification of large data sets composed of 2-D images. However, due to the existence of noisy or correlated spectral bands in the spectral domain and nonuniform pixels in the spatial neighborhood, HSI classification results are often degraded and unacceptable. However, the elementary CNN models often find intrinsic representation of pattern directly when employed to explore the HSI in the spectral-spatial domain. In this article, we design an end-to-end spectral-spatial squeeze-and-excitation (SE) residual bag-of-feature (S3EResBoF) learning framework for HSI classification that takes as input raw 3-D image cubes without engineering and builds a codebook representation of transform feature by motivating the feature maps facilitating classification by suppressing useless feature maps based on patterns present in the feature maps. To boost the classification performance and learn the joint spatial-spectral features, every residual block is connected to every other 3-D convolutional layer through an identity mapping followed by an SE block, thereby facilitating the rich gradients through backpropagation. Additionally, we introduce batch normalization on every convolutional layer (ConvBN) to regularize the convergence of the network and scale invariant BoF quantization for the measure of classification. The experiments conducted using three well-known HSI data sets and compared with the state-of-the-art classification methods reveal that S3EResBoF provides competitive performance in terms of both classification and computation time.

A Multiscale Deep Learning Approach for High-Resolution Hyperspectral Image Classification

Hyperspectral imagery (HSI) has emerged as a highly successful sensing modality for a variety of applications ranging from urban mapping to environmental monitoring and precision agriculture. Despite the efforts made by the scientific community, developing reliable algorithms of HSI classification remains a challenging problem, especially for high-resolution HSI data where there is often larger intraclass variability combined with a scarcity of the ground-truth data and class imbalance. In recent years, deep neural networks have emerged as a promising strategy for problems of HSI classification where they have shown remarkable potential for learning joint spectral-spatial features efficiently via backpropagation. In this letter, we propose a deep learning strategy for HSI classification that combines different convolutional neural networks, especially designed to efficiently learn joint spatial-spectral features over multiple scales. Our method achieves an overall classification accuracy of 66.73% on the 2018 IEEE GRSS hyperspectral data set-a high-resolution data set that includes 20 urban land-cover and land-use classes.

HybridSN: Exploring 3-D–2-D CNN Feature Hierarchy for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is widely used for the analysis of remotely sensed images. Hyperspectral imagery includes varying bands of images. Convolutional Neural Network (CNN) is one of the most frequently used deep learning based methods for visual data processing. The use of CNN for HSI classification is also visible in recent works. These approaches are mostly based on 2D CNN. Whereas, the HSI classification performance is highly dependent on both spatial and spectral information. Very few methods have utilized the 3D CNN because of increased computational complexity. This letter proposes a Hybrid Spectral Convolutional Neural Network (HybridSN) for HSI classification. Basically, the HybridSN is a spectral-spatial 3D-CNN followed by spatial 2D-CNN. The 3D-CNN facilitates the joint spatial-spectral feature representation from a stack of spectral bands. The 2D-CNN on top of the 3D-CNN further learns more abstract level spatial representation. Moreover, the use of hybrid CNNs reduces the complexity of the model compared to 3D-CNN alone. To test the performance of this hybrid approach, very rigorous HSI classification experiments are performed over Indian Pines, Pavia University and Salinas Scene remote sensing datasets. The results are compared with the state-of-the-art hand-crafted as well as end-to-end deep learning based methods. A very satisfactory performance is obtained using the proposed HybridSN for HSI classification. The source code can be found at \url{https://github.com/gokriznastic/HybridSN}.

CNN-Based Multilayer Spatial–Spectral Feature Fusion and Sample Augmentation With Local and Nonlocal Constraints for Hyperspectral Image Classification

The extraction of joint spatial-spectral features has been proved to improve the classification performance of hyperspectral images (HSIs). Recently, utilizing convolutional neural networks (CNNs) to learn joint spatial-spectral features has become of great interest. However, the existing CNN models ignore complementary spatial-spectral information among the shallow and deep layers. Moreover, insufficient training samples in HSIs afflict these CNN models with overfitting problem. In order to address these problems, a novel CNN method for HSI classification is proposed. It considers multilayer spatial-spectral feature fusion and sample augmentation with local and nonlocal constraints, which is abbreviated as MSLN-CNN. In MSLN-CNN, a triple-architecture CNN is constructed to extract spatial-spectral features by cascading spectral features to dual-scale spatial features from shallow to deep layers. Then, multilayer spatial-spectral features are fused to learn complementary information among the shallow layers with detailed information and the deep layers with semantic information. Finally, the multilayer spatial-spectral feature fusion and classification are integrated into a unified network, and MSLN-CNN can be optimized in the end-to-end way. To alleviate the small sample size problem, the unlabeled samples having high confidences on local spatial constraint and nonlocal spectral constraint are selected and prelabeled. The nonlocal spectral constraint considers the structure information with spectrally similar samples in the nonlocal searching, while the local spatial constraint utilizes the contextual information with spatially adjacent samples. Experimental results on several hyperspectral datasets demonstrate that the proposed method achieves more encouraging classification performance than the current state-of-the-art classification methods, especially with the limited training samples.

Convolutional neural network in network (CNNiN): hyperspectral image classification and dimensionality reduction

Classification is a principle technique in hyperspectral images (HSIs), where a label is assigned to each pixel based on its characteristics. However, due to lack of labelled training instances in HSIs and also its ultra-high dimensionality, deep learning approaches need a special consideration for HSI classification. As one of the first works in the HSI classification, this study proposes a novel network pipeline called convolutional neural network in network (which is deeper than the existing approaches) by jointly utilising the spatial and spectral information and produces high-level features from the original HSI. This can occur by using spatial-spectral relationships of individual pixel vector at the initial component of the proposed pipeline; the extracted features are then combined to form a joint spatial-spectral feature map. Finally, a recurrent neural network is trained on the extracted features which contain wealthy spectral and spatial properties of the HSI to predict the corresponding label of each vector. The model has been tested on two large scale hyperspectral datasets in terms of classification accuracy, training error, and computational time.

Joint Spatial-Spectral Feature Space Clustering for Speech Activity Detection from ECoG Signals

Brain-machine interfaces for speech restoration have been extensively studied for more than two decades. The success of such a system will depend in part on selecting the best brain recording sites and signal features corresponding to speech production. The purpose of this study was to detect speech activity automatically from electrocorticographic signals based on joint spatial-frequency clustering of the ECoG feature space. For this study, the ECoG signals were recorded while a subject performed two different syllable repetition tasks. We found that the optimal frequency resolution to detect speech activity from ECoG signals was 8Hz, achieving 98.8% accuracy by employing support vector machines as a classifier. We also defined the cortical areas that held the most information about the discrimination of speech and nonspeech time intervals. Additionally, the results shed light on the distinct cortical areas associated with the two syllables repetition tasks and may contribute to the development of portable ECoG-based communication.