Classification Of Hyperspectral Data Research Articles

Interactive transformer and CNN network for fusion classification of hyperspectral and LiDAR data

ABSTRACT The Transformer has become pivotal for the integrated analysis of multi-source remote-sensing (RS) data in Earth observation, particularly in applications such as the fusion classification of hyperspectral images (HSI) and Light Detection and Ranging (LiDAR) data. However, Transformers are often employed as effective feature extractors by adopting similar processing blocks for different modalities from multi-source sensors, overlooking differences in imaging principles and data characteristics. Moreover, in the feature extraction process across different sensor data, there is a lack of necessary cross-modal information interaction, leading to insufficient utilization of complementary information between different sensors and resulting in suboptimal fusion outcomes. In this paper, we propose an interactive Transformer and CNN network for the fusion classification of HSI and LiDAR data. Specifically, a heterogeneous three-branch network architecture is designed for HSI and LiDAR data, where Transformers and CNNs encapsulate global contextual spatial and spectral information for HSI and capture geometric elevation patterns for LiDAR data, respectively. Elevation-Spatial Interaction (ESI) and Spectral-Spatial Interaction (SSI) modules are then introduced for multi-stage feature interaction. ESI enables the CNN-Transformer network to focus on essential local elevation details while simultaneously modelling global contextual spatial information. SSI facilitates the Transformer-Transformer network to cyclically intertwine spectral and spatial information for long-range spectral-spatial feature fusion. Finally, the interacted elevation, spatial, and spectral features undergo the Gated Fusion module to achieve hierarchical fusion adaptively, resulting in an elevation-spatial-spectral representation. Experiments conducted on three benchmark HSI-LiDAR datasets demonstrate the effectiveness of our proposed approach.

Classification of hyperspectral and LiDAR data by transformer-based enhancement

ABSTRACT The integration of multi-modal data allows for a more accurate representation of the ground characteristics. For a comprehensive interpretation of remote sensing data, existing multi-modal data fusion research mainly focuses on the joint utilization of 3D Light Detection and Ranging (LiDAR) and 2D Hyperspectral Image (HSI) data. However, existing algorithms do not pay much attention to the interaction of high-level semantic information between different modal data before fusion. This paper proposes a novel multi-modal data fusion deep learning network with the Cross-Modal Self-Attentive Feature Fusion Transformer (SAFFT). The framework employs a multi-head self-attention layer to fuse various attention information from multiple heads, effectively enhancing advanced feature information from different modalities for comprehensive integration. Experimental results on the Houston 2013 dataset demonstrate the effectiveness of the proposed method, which achieves an overall accuracy (OA) of 94.3757% in classifying 15 semantic classes.

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Multi-level interactive fusion network based on adversarial learning for fusion classification of hyperspectral and LiDAR data

With the growing awareness of the limitations of unimodal data, research on joint classification using multimodal remote sensing data has garnered increasing attention. However, existing methods still have certain deficiencies in dealing with feature extraction and feature fusion of multimodal remote sensing data. In order to obtain more accurate classification results, it is crucial to fully exploit and utilize the complementary information in multimodal remote sensing data. In this paper, a multi-level interactive fusion network based on adversarial learning (MLIF-AL) is proposed for hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion classification. First, the feature extraction part based on generative adversarial network (GAN) utilizes adversarial training between the generator and the discriminator to guide the network to learn more discriminative and distinguished feature representations. Meanwhile, a cross-modal interactive information extraction (CMIIE) module is also designed for the generator encoding part to promote the interaction between multimodal data from a global perspective and realize the full utilization of multimodal complementary information. In addition, in the multi-level feature fusion classification (MLFFC) part, the complementarities between the features learned by the neural networks at each layer are comprehensively utilized to obtain higher-level multimodal fusion semantic information. Finally, the adversarial loss and classification loss are combined for network training. Extensive experiments on three commonly used HSI and LiDAR datasets and comparisons with state-of-the-art methods demonstrate the effectiveness and superiority of the model.

DCFF-Net: Deep Context Feature Fusion Network for High-Precision Classification of Hyperspectral Image

Hyperspectral images (HSI) contain abundant spectral information. Efficient extraction and utilization of this information for image classification remain prominent research topics. Previously, hyperspectral classification techniques primarily relied on statistical attributes and mathematical models of spectral data. Deep learning classification techniques have recently been extensively utilized for hyperspectral data classification, yielding promising outcomes. This study proposes a deep learning approach that uses polarization feature maps for classification. Initially, the polar co-ordinate transformation method was employed to convert the spectral information of all pixels in the image into spectral feature maps. Subsequently, the proposed Deep Context Feature Fusion Network (DCFF-NET) was utilized to classify these feature maps. The model was validated using three open-source hyperspectral datasets: Indian Pines, Pavia University, and Salinas. The experimental results indicated that DCFF-NET achieved excellent classification performance. Experimental results on three public HSI datasets demonstrated that the proposed method accurately recognized different objects with an overall accuracy (OA) of 86.68%, 94.73%, and 95.14% based on the pixel method, and 98.15%, 99.86%, and 99.98% based on the pixel-patch method.

Multi-Feature Cross Attention-Induced Transformer Network for Hyperspectral and LiDAR Data Classification

Transformers have shown remarkable success in modeling sequential data and capturing intricate patterns over long distances. Their self-attention mechanism allows for efficient parallel processing and scalability, making them well-suited for the high-dimensional data in hyperspectral and LiDAR imagery. However, further research is needed on how to more deeply integrate the features of two modalities in attention mechanisms. In this paper, we propose a novel Multi-Feature Cross Attention-Induced Transformer Network (MCAITN) designed to enhance the classification accuracy of hyperspectral and LiDAR data. The MCAITN integrates the strengths of both data modalities by leveraging a cross-attention mechanism that effectively captures the complementary information between hyperspectral and LiDAR features. By utilizing a transformer-based architecture, the network is capable of learning complex spatial-spectral relationships and long-range dependencies. The cross-attention module facilitates the fusion of multi-source data, improving the network’s ability to discriminate between different land cover types. Extensive experiments conducted on benchmark datasets demonstrate that the MCAITN outperforms state-of-the-art methods in terms of classification accuracy and robustness.

Joint Classification of Hyperspectral and LiDAR Data Based on Adaptive Gating Mechanism and Learnable Transformer

Utilizing multi-modal data, as opposed to only hyperspectral image (HSI), enhances target identification accuracy in remote sensing. Transformers are applied to multi-modal data classification for their long-range dependency but often overlook intrinsic image structure by directly flattening image blocks into vectors. Moreover, as the encoder deepens, unprofitable information negatively impacts classification performance. Therefore, this paper proposes a learnable transformer with an adaptive gating mechanism (AGMLT). Firstly, a spectral–spatial adaptive gating mechanism (SSAGM) is designed to comprehensively extract the local information from images. It mainly contains point depthwise attention (PDWA) and asymmetric depthwise attention (ADWA). The former is for extracting spectral information of HSI, and the latter is for extracting spatial information of HSI and elevation information of LiDAR-derived rasterized digital surface models (LiDAR-DSM). By omitting linear layers, local continuity is maintained. Then, the layer Scale and learnable transition matrix are introduced to the original transformer encoder and self-attention to form the learnable transformer (L-Former). It improves data dynamics and prevents performance degradation as the encoder deepens. Subsequently, learnable cross-attention (LC-Attention) with the learnable transfer matrix is designed to augment the fusion of multi-modal data by enriching feature information. Finally, poly loss, known for its adaptability with multi-modal data, is employed in training the model. Experiments in the paper are conducted on four famous multi-modal datasets: Trento (TR), MUUFL (MU), Augsburg (AU), and Houston2013 (HU). The results show that AGMLT achieves optimal performance over some existing models.

A novel graph-attention based multimodal fusion network for joint classification of hyperspectral image and LiDAR data

The joint classification of hyperspectral image (HSI) and Light Detection and Ranging (LiDAR) data can provide complementary information for each other, which has become a prominent topic in the field of remote sensing. Nevertheless, the common CNN-based fusion techniques still suffer from the following drawbacks. (1) Most of these models omit the correlation and complementarity between different data sources and always fail to model the long-distance dependencies of spectral information well. (2) Simply splicing the multi-source feature embeddings overlooks the deep semantic relationships among them. To tackle these issues, we propose a novel graph-attention based multimodal fusion network (GAMF). Specifically, it employs three major components, including an HSI–LiDAR feature extractor, a graph-attention based fusion module and a classification module. In the feature extraction module, we consider the correlation and complementarity between multi-sensor data by parameter sharing and employ Gaussian tokenization for feature transformation additionally. To address the problem of long-distance dependencies, the deep fusion module utilizes modality-specific tokens to construct an undirected weighted graph, which is essentially a heterogeneous graph. And the deep semantic relationships between them are exploited utilizing a graph-attention based fusion framework. At the end, two fully connected layers classify the fused embeddings. Experiment evaluations on several benchmark HSI–LiDAR datasets (Trento, University of Houston 2013 and MUUFL) show that GAMF achieves more accurate prediction results than some state-of-the-art baselines. The code is available at https://github.com/tyust-dayu/GAMF.

Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

In the face of complex scenarios, the information insufficiency of classification tasks dominated by a single modality has led to a bottleneck in classification performance. The joint application of multimodal remote sensing data for surface observation tasks has garnered widespread attention. However, issues such as sample differences between modalities and the lack of correlation in physical features have limited the performance of classification tasks. Establishing effective interaction between multimodal data has become another significant challenge. To fully integrate heterogeneous information from multiple modalities and enhance classification performance, this paper proposes a dual-branch cross-Transformer feature fusion network aimed at joint land cover classification of hyperspectral imagery (HSI) and Light Detection and Ranging (LiDAR) data. The core idea is to leverage the potential of convolutional operators to represent spatial features, combined with the advantages of the Transformer architecture in learning remote dependencies. The framework employs an improved self-attention mechanism to aggregate features within each modality, highlighting the spectral information of HSI and the spatial (elevation) information of LiDAR. The feature fusion module based on cross-attention integrates deep features from two modalities, achieving complementary information through cross-modal attention. The classification task is performed using jointly obtained spectral and spatial features. Experiments were conducted on three multi-source remote sensing classification datasets, demonstrating the effectiveness of the proposed model compared to existing methods.

Interactive Enhanced Network Based on Multihead Self-Attention and Graph Convolution for Classification of Hyperspectral and LiDAR Data

Interactive Enhanced Network Based on Multihead Self-Attention and Graph Convolution for Classification of Hyperspectral and LiDAR Data

Spectral classification of AVIRIS NG hyperspectral data for discriminating coastal foredunes based on vegetation species: a case study from Cuddalore district of Tamil Nadu, South India

Spectral classification of AVIRIS NG hyperspectral data for discriminating coastal foredunes based on vegetation species: a case study from Cuddalore district of Tamil Nadu, South India

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

Collaborative classification of hyperspectral and LiDAR data based on CNN-transformer

Collaborative classification of hyperspectral and LiDAR data based on CNN-transformer

Multimodal Deep Learning for Semisupervised Classification of Hyperspectral and LiDAR Data

Multimodal Deep Learning for Semisupervised Classification of Hyperspectral and LiDAR Data

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning

Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

Pseudo-Labelling Contrastive Learning for Semi-supervised Hyperspectral and LiDAR Data Classification

Pseudo-Labelling Contrastive Learning for Semi-supervised Hyperspectral and LiDAR Data Classification

Attention-Guided Fusion and Classification for Hyperspectral and LiDAR Data

The joint use of hyperspectral image (HSI) and Light Detection And Ranging (LiDAR) data has been widely applied for land cover classification because it can comprehensively represent the urban structures and land material properties. However, existing methods fail to combine the different image information effectively, which limits the semantic relevance of different data sources. To solve this problem, in this paper, an Attention-guided Fusion and Classification framework based on Convolutional Neural Network (AFC-CNN) is proposed to classify the land cover based on the joint use of HSI and LiDAR data. In the feature extraction module, AFC-CNN employs the three dimensional convolutional neural network (3D-CNN) combined with a multi-scale structure to extract the spatial-spectral features of HSI, and uses a 2D-CNN to extract the spatial features from LiDAR data. Simultaneously, the spectral attention mechanism is adopted to assign weights to the spectral channels, and the cross attention mechanism is introduced to impart significant spatial weights from LiDAR to HSI, which enhance the interaction between HSI and LiDAR data and leverage the fusion information. Then two feature branches are concatenated and transferred to the feature fusion module for higher-level feature extraction and fusion. In the fusion module, AFC-CNN adopts the depth separable convolution connected through the residual structures to obtain the advanced features, which can help reduce computational complexity and improve the fitting ability of the model. Finally, the fused features are sent into the linear classification module for final classification. Experimental results on three datasets, i.e., Houston, MUUFL and Trento datasets show that the proposed AFC-CNN framework achieves better classification accuracy compared with the state-of-the-art algorithms. The overall accuracy of AFC-CNN on Houston, MUUFL and Trento datasets are 94.2%, 95.3% and 99.5%, respectively.

Band selection using hybridization of particle swarm optimization and crow search algorithm for hyperspectral data classification

Band selection using hybridization of particle swarm optimization and crow search algorithm for hyperspectral data classification