Spatial Spectral Information Research Articles

Semantic-Guided Transformer Network for Crop Classification in Hyperspectral Images

The hyperspectral remote sensing images of agricultural crops contain rich spectral information, which can provide important details about crop growth status, diseases, and pests. However, existing crop classification methods face several key limitations when processing hyperspectral remote sensing images, primarily in the following aspects. First, the complex background in the images. Various elements in the background may have similar spectral characteristics to the crops, and this spectral similarity makes the classification model susceptible to background interference, thus reducing classification accuracy. Second, the differences in crop scales increase the difficulty of feature extraction. In different image regions, the scale of crops can vary significantly, and traditional classification methods often struggle to effectively capture this information. Additionally, due to the limitations of spectral information, especially under multi-scale variation backgrounds, the extraction of crop information becomes even more challenging, leading to instability in the classification results. To address these issues, a semantic-guided transformer network (SGTN) is proposed, which aims to effectively overcome the limitations of these deep learning methods and improve crop classification accuracy and robustness. First, a multi-scale spatial–spectral information extraction (MSIE) module is designed that effectively handle the variations of crops at different scales in the image, thereby extracting richer and more accurate features, and reducing the impact of scale changes. Second, a semantic-guided attention (SGA) module is proposed, which enhances the model’s sensitivity to crop semantic information, further reducing background interference and improving the accuracy of crop area recognition. By combining the MSIE and SGA modules, the SGTN can focus on the semantic features of crops at multiple scales, thus generating more accurate classification results. Finally, a two-stage feature extraction structure is employed to further optimize the extraction of crop semantic features and enhance classification accuracy. The results show that on the Indian Pines, Pavia University, and Salinas benchmark datasets, the overall accuracies of the proposed model are 98.24%, 98.34%, and 97.89%, respectively. Compared with other methods, the model achieves better classification accuracy and generalization performance. In the future, the SGTN is expected to be applied to more agricultural remote sensing tasks, such as crop disease detection and yield prediction, providing more reliable technical support for precision agriculture and agricultural monitoring.

Burned-area sub-pixel mapping based on spatial–spectral information at super-pixel scale for multi-spectral image

ABSTRACT Due to the complex environments of burned areas and limitations of hardware devices, the collected multi-spectral image (MSI) is sometimes with many mixed pixels to hinder the accurate mapping of burned areas. To solve this problem, sub-pixel mapping (SPM) technology has been applied to handle with these mixed pixels to map burned areas. However, the spatial–spectral information of burned areas used by SPM is usually constructed in a specified rectangular local window, and the number of spectral bands utilized by SPM is also little, affecting the mapping accuracy of burned areas. To improve the mapping accuracy of burned areas, we propose burned-area SPM based on spatial–spectral information at super-pixel scale for multi-spectral image (SSIASC). In SSIASC, super-pixels representing the burned areas with irregular distribution are obtained by interpolation and then segmentation of the original coarse MSI. The extended random walker algorithm is then used to calculate the spatial correlation in super-pixels to obtain spatial term, and at the same time the normalized model is constructed to calculate all the spectral bands in super-pixels to yield spectral term. Next, the two terms are integrated to produce the objective term with spatial–spectral information at super-pixel scale. Finally, particle swarm optimization is employed to optimize the objective term to derive the burned-area mapping result. Experimental results on the two burned-area MSIs show that the proposed SSIASC produces the better results than the state-of-the-art SPM methods.

UVMSR: a novel approach to hyperspectral image super-resolution by fusing U-Net and Mamba

ABSTRACT The hyperspectral image super resolution (HSISR) task has been thoroughly researched and has shown notable advancements. However, existing deep neural network-based methods for HSISR face challenges in effectively utilizing global spectral-spatial information. While transformer-based models exhibit strong global modelling capabilities, their high computational complexity poses a challenge when applied to hyperspectral image processing. Recently, state space models (SSM) with efficient hardware-aware design, such as Mamba, have demonstrated promising capabilities for long sequence modelling. In this study, we introduce a HSISR method (UVMSR) that combines U-Net and Mamba. UVMSR is a hybrid CNN-SSM module that integrates the local feature extraction capabilities of convolutional layers with the long-range dependency capturing abilities of SSMs. Specifically, we design the U-Net network structure for HSISR and apply V-Mamba within it for global modelling to capture spectral-spatial features. V-Mamba utilizes positional embedding to label the image sequences and employs a bidirectional state-space model for global context modelling. Additionally, a spectral-spatial feature expansion (SSFE) module is designed for better recovery of detailed information in hyperspectral images during the up-sampling process of U-Net. This paper evaluates the performance of UVMSR on the Chikusei, Pavia Centre, Houston 2018 and Cave datasets. The results of the comparison with other state-of-the-art methods demonstrate that UVMSR outperforms them, achieving unparalleled performance in reconstruction results. The code is available at https://github.com/TeresaTing/UVMSR.

Discrete Cosine Transform-Based Joint Spectral–Spatial Information Compression and Band-Correlation Calculation for Hyperspectral Feature Extraction

Prediction tasks over pixels in hyperspectral images (HSI) require careful effort to engineer the features used for learning a classifier. However, the generated classification map may suffer from an over-smoothing problem, which is manifested in significant differences from the original image in terms of object boundaries and details. To address this over-smoothing problem, we designed a method for extracting spectral–spatial-band-correlation (SSBC) features. In SSBC features, joint spectral–spatial feature extraction is considered a discrete cosine transform-based information compression, where a flattening operation is used to avoid the high computational cost induced by the requirement of distillation from 3D images for joint spectral–spatial information. However, this process can yield extracted features with lost spectral information. We argue that increasing the spectral information in the extracted features is the key to addressing the over-smoothing problem in the classification map. Consequently, the normalized difference vegetation index and iron oxide are improved for HSI data in extracting band-correlation features as added spectral information because their calculations, involving two spectral bands, are not appropriate for the abundant spectral bands of HSI. Experimental results on four real HSI datasets show that the proposed features can significantly mitigate the over-smoothing problem, and the classification performance is comparable to that of state-of-the-art deep features.

Tensor-Based Few-Shot Learning for Cross-Domain Hyperspectral Image Classification

Few-shot learning (FSL) is an effective solution for cross-domain hyperspectral image (HSI) classification, which could address the limited labeled samples of the target domain. Current FSL methods mostly utilize the 3D-CNN to transform the spatial and spectral information into a single feature to model an HSI, which means that spatial and spectral information are treated equally in the feature-modeling process. However, spectral information is considered to be more domain-invariant than spatial information. Treating the spatial and spectral information equally may result in parameter redundancy and undesirable cross-domain classification performance. In this paper, we attempt to use tensor mathematics for modeling the HSI and propose a novel few-shot learning method, called tensor-based few-shot Learning (TFSL) for cross-domain HSI classification, which aims to guide the model to focus on the extraction of domain-invariant spectral dependencies. Specifically, we first propose a spatial–spectral tensor decomposition (SSTD) model to provide a mathematical explanation of the input HSI, representing the local spatial–spectral information as 1D and 2D local tensors to reduce the data redundancy. Additionally, a tensor-based hybrid two-stream (THT) model is proposed for extracting the domain-invariant spatial–spectral tensor feature by using 1D-CNN and 2D-CNN. Furthermore, we adopt a 1D-CNN tensor feature enhancement block to enhance the spectral feature of hybrid two-stream tensors and guide the THT model to concentrate on the modeling of spectral dependencies. Finally, the proposed TFSL is evaluated on four public HSI datasets, and the extensive experimental results demonstrate that the proposed TFSL significantly outperforms other advanced FSL methods.

Cascaded hybrid convolutional autoencoder network for spectral-spatial nonlinear hyperspectral unmixing

ABSTRACT Due to the complex interaction of photons between materials, linear hyperspectral unmixing (HU) is limited in real scenarios, making nonlinear unmixing a promising alternative. With the advancement of deep learning (DL), nonlinear unmixing methods based on the convolutional autoencoder (CAE) have gained considerable traction in HU. However, these unmixing methods struggle to integrate spectral and spatial information while reducing the loss of material details during the unmixing process. Therefore, we propose a cascaded hybrid CAE nonlinear unmixing network, called CHCANet, which effectively leverages convolutional combinations to deeply explore the spectral-spatial information from hyperspectral data and preserve the material details through self-perception. Specifically, each CAE in CHCANet combines 1-D and 2-D convolutions, fully utilizing the flexibility and simplicity of 1-D convolutions to capture spectral features and the spatial correlation handling capability of the 2-D convolution. Moreover, we apply the self-perception mechanism to the nonlinear HU task, which can establish the cycle consistency of the network, strengthen mutual connections between encoders, and effectively preserve high-level semantic information. Following this, the optimized self-perception loss further enhances CHCANet’s perception capability of nonlinear components and strengthens the connection between the decoder directly associated with image reconstruction. Extensive experiments on synthetic and real datasets demonstrate the effectiveness of CHCANet and show excellent competitiveness compared to state-of-the-art unmixing methods.

MultiScale spectral–spatial convolutional transformer for hyperspectral image classification

AbstractDue to the powerful ability in capturing the global information, transformer has become an alternative architecture of CNNs for hyperspectral image classification. However, general transformer mainly considers the global spectral information while ignores the multiscale spatial information of the hyperspectral image. In this paper, we propose a multiscale spectral–spatial convolutional transformer (MultiFormer) for hyperspectral image classification. First, the developed method utilizes multiscale spatial patches as tokens to formulate the spatial transformer and generates multiscale spatial representation of each band in each pixel. Second, the spatial representation of all the bands in a given pixel are utilized as tokens to formulate the spectral transformer and generate the multiscale spectral–spatial representation of each pixel. Besides, a modified spectral–spatial CAF module is constructed in the MultiFormer to fuse cross‐layer spectral and spatial information. Therefore, the proposed MultiFormer can capture the multiscale spectral–spatial information and provide better performance than most of other architectures for hyperspectral image classification. Experiments are conducted over commonly used real‐world datasets and the comparison results show the superiority of the proposed method.

MBSFC: hyperspectral image classification based on multi-branch and spectral feature conversion

ABSTRACT Hyperspectral image classification (HSI) is a vital aspect of remote sensing technology. However, the high-dimensional nature of HSI and the continuous spectral bands significantly impact its classification performance. In this paper, we propose a classification method called MBSFC for HSI classification. MBSFC utilizes multi-branch and spectral feature conversion (SFC) techniques to eliminate redundant information and extract valuable features. Firstly, to preserve spectral-spatial information, the SFC block is introduced to eliminate redundant information through up-sampling. Subsequently, the feature maps from the SFC block and the spectral bands reduced by a factor of four are fed into three branches: the spectral branch, spatial-X branch, and spatial-Y branch. Each branch employs a 3D-CNN-based dense block and attention mechanism to extract useful features. Finally, the obtained features from the three branches are fused for HSI classification. To cater to different application scenarios, we divide MBSFC into three models with different network structures: MBSFC-s, MBSFC-m, and MBSFC-l. The MBSFC-l model, with a three-branch structure, achieves the best performance. The three models differ in the number of branches in the feature extraction. Experimental results on four publicly available hyperspectral datasets show that MBSFC achieves competitive results with small samples compared to other state-of-the-art methods. The code is available at https://github.com/TeresaTing/MBSFC.

An efficient hyperspectral image classification method using retentive network

In recent computer vision tasks, the vision transformer (ViT) has demonstrated competitive ability. However, ViT still has problems: the computational complexity of the self-attention layer leads to expensive and slow interference, and processing all tokens for high-resolution images may slow down due to the layer’s quadratic complexity. Recently, a retentive network with excellent performance, training parallelism, and an inexpensive inference cost was proposed. For hyperspectral image (HSI) classification, this paper proposesa retention-based network model called the HSI retentive network (HSIRN). The proposed model allows memory usage independent of the token’s sequence, facilitating the efficient processing of high-resolution images with low inference and computational costs. Although the retention encoder can extract global data, it pays limited attention to local data. A powerful tool for extracting local information is a convolutional neural network (CNN). The proposed HSIRN model uses a specific CNN-based block to extract local spectral-spatial information. To maintain degradation between successive vertical and horizontal positions with the depth dimension of the HSI, we propose a three-dimensional retention mechanism for the three-dimensional HSI dataset in the retention encoder. By efficiently using both local and global spectral-spatial information, the proposed method offers a potent tool for HSI classification. We evaluated the classification performance of the proposed HSIRN approach on four datasets through comprehensive examinations, and the results demonstrated its superiority over state-of-the-art methods. At https://github.com/RajatArya22/HSIRN, the source code will be available to the public.

MDSCNN: Remote Sensing Image Spatial–Spectral Fusion Method via Multi-Scale Dual-Stream Convolutional Neural Network

Pansharpening refers to enhancing the spatial resolution of multispectral images through panchromatic images while preserving their spectral features. However, existing traditional methods or deep learning methods always have certain distortions in the spatial or spectral dimensions. This paper proposes a remote sensing spatial–spectral fusion method based on a multi-scale dual-stream convolutional neural network, which includes feature extraction, feature fusion, and image reconstruction modules for each scale. In terms of feature fusion, we propose a multi cascade module to better fuse image features. We also design a new loss function aim at enhancing the high degree of consistency between fused images and reference images in terms of spatial details and spectral information. To validate its effectiveness, we conduct thorough experimental analyses on two widely used remote sensing datasets: GeoEye-1 and Ikonos. Compared with the nine leading pansharpening techniques, the proposed method demonstrates superior performance in multiple key evaluation metrics.

Lightweight omni-dimensional dynamic convolution with spatial-spectral self-attention network for hyperspectral image classification

ABSTRACT Omni-dimensional dynamic convolution can adjust spatial sizes, the amount of input/output channels, and kernel size based on the characteristics of the input, thus improving the capacity to extract characteristics through convolution, resulting in increased computation and model complexity. As a result, it is challenging to deploy these models on resource-limited airborne high-spectral testing systems. This paper proposes a lightweight omni-dimensional dynamic convolution network as a possible solution to this issue. In this network, the LS2CM module is used for spatial-spectral information extraction, and the pyramidal residual network is the basic architecture for extracting features from shallow to deep layers, and lightweight omni-dimensional dynamic convolution is adopted in the convolution module. Overcoming the pyramidal structure can only extract local information, we proposed a new simple spatial-spectral self-attention mechanism to extract global information from hyperspectral. Besides, the Huber function as an activation function. The IP, WHU-Hi-LongKou, SA, and PU datasets were used to verify the accuracy of the suggested network model. The results show that the proposed lightweight network maintains an appropriate equilibrium among model complexity as well as accuracy.

Coupled Spatial-Spectral Constrained Convolutional Fusion Network for Hyperspectral and Panchromatic images

ABSTRACT Target monitoring is an important subject in machine vision. Hyperspectral image (HSI) can effectively assist the target detection and recognition effect of traditional optical images because of its rich spectral information. However, limited by pixel mixing, the resolution of HSI is generally lower than that of optical image, which restricts the monitoring distance and accuracy. Therefore, a fusion method of HSI and panchromatic image (PAN) based on coupled spatial-spectral constrained convolution neural network is proposed in this paper to improve the spatial resolution of HSI and reduce the spectral distortion. Through this approach, the linear spectral mixing model and the spatial-spectral transformation constraint model are incorporated into the learning stage of the coupled convolutional neural network, aiming to make full use of the spatial-spectral information of HSI and PAN, and improve the spectral fidelity of fused images. Experiments on several groups of HSI and PAN data sets show that compared with some currently proposed HSI and PAN fusion methods, the proposed approach has better spectral fidelity and lower fusion errors, so as to improve the monitoring distance and accuracy of machine vision in engineering applications and expand the engineering application scenarios.

Hierarchical homogeneity-based superpixel segmentation: application to hyperspectral image analysis

ABSTRACT Hyperspectral image (HI) analysis approaches have recently become increasingly complex and sophisticated. Recently, the combination of spectral-spatial information and superpixel techniques have addressed some hyperspectral data issues, such as the higher spatial variability of spectral signatures and dimensionality of the data. However, most existing superpixel approaches do not account for specific HI characteristics resulting from its high spectral dimension. In this work, we propose a multiscale superpixel method that is computationally efficient for processing hyperspectral data. The Simple Linear Iterative Clustering (SLIC) oversegmentation algorithm, on which the technique is based, has been extended hierarchically. Using a novel robust homogeneity testing, the proposed hierarchical approach leads to superpixels of variable sizes but with higher spectral homogeneity when compared to the classical SLIC segmentation. For validation, the proposed homogeneity-based hierarchical method was applied as a preprocessing step in the spectral unmixing and classification tasks carried out using, respectively, the Multiscale sparse Unmixing Algorithm (MUA) and the CNN-Enhanced Graph Convolutional Network (CEGCN) methods. Simulation results with both synthetic and real data show that the technique is competitive with state-of-the-art solutions.

LatLBP: Spatial-spectral latent local binary pattern for hyperspectral image classification

The rich spatial and spectral information brings great potential for pixel-wise classification of hyperspectral image (HSI). Recently, Local Binary Pattern (LBP) as a prominent texture operator has been introduced for discriminative feature extraction of complex volumetric HSI. However, the popular two-dimensional LBP series can only capture spatial structure features and lack pattern descriptions of the spectral domain. The extended three-dimensional LBP series are capable of local multidimensional descriptions, while the information mining for narrow and continuous bands in the spectral domain is still limited. To tackle these issues, we develop a novel local binary pattern named LatLBP for investigating spatial-spectral latent information, which provides a low-dimensional, fine-grained, high-level representation of dual-domain latent semantic information by exploring the group structure of data. LatLBP consists of representative spatial latent (RSLat) LBP encoding and grouped spectral latent (GSLat) LBP encoding, where RSLat describes the local spatial potential features of each grouped representative band, and GSLat characterizes information about each group of latent spectral factors. Besides, a supporting neighborhood band grouping (NBG) algorithm is also designed to provide a finer group structure. Extensive experiments conducted on four HSI datasets indicate that the optimal classification results, i.e., 86.94% for Pavia University, 88.66% for Indian Pines, 77.75% for HanChuan, and 85.54% for HongHu, are achieved by either the proposed LatLBP or GSLat compared to competitive operators.

SSCDN: a spatial-spectral collaborative network for hyperspectral image denoising.

Hyperspectral imaging provides the full spectrum at each point of the whole field-of-view, and thus is being extensively employed in remote sensing, surveillance, medical diagnostics and biological research. However, the intrinsically limited photons for each spectral band and the inevitable noise during acquisition result in complex degradation of hyperspectral images (HSIs) that adversely impacts the subsequent data analysis. Yet, it remains challenging for current HSI denoising methods to effectively address HSI datasets that are significantly contaminated by complex noise, especially in terms of spectral recovery. In this paper, we propose a spatial-spectral collaborative denoising network (SSCDN) that makes full use of spatial-spectral correlation information for HSI denoising. Through the combination of attention mechanism and specifically designed spatial-spectral collaborative attention module along with a multi-loss joint optimization strategy, the proposed model achieves superior denoising performance while well-preserving spectral and spatial features for complex degradation. Extensive experimental results on simulated and real data for remote sensing and biomedical applications demonstrate that the proposed SSCDN outperforms other state-of-the-art competitive HSI denoising methods under various noise settings, especially in terms of structural-spectral fidelity and the model robustness against noise.

Single-pixel-based hyperspectral microscopy

Hyperspectral imaging allows to collect both spatial and quasi-continuous spectral information of an object. This work shows the innovative combination of single-pixel microscopy with hyperspectral imaging. An affordable hyperspectral microscope is able to observe micrometer-scale features of inorganic and biological samples and to reconstruct their spectral distribution with a high accuracy (i.e., a spatial and a spectral resolution of 9.0 μm and of 2.1 nm in the visible range, respectively). Furthermore, a statistical algorithm enables the identification of spectral responses of the targeted features as well as their classification.

MGCET: MLP-mixer and Graph Convolutional Enhanced Transformer for Hyperspectral Image Classification

The vision transformer (ViT) has demonstrated performance comparable to that of convolutional neural networks (CNN) in the hyperspectral image classification domain. This is achieved by transforming images into sequence data and mining global spectral-spatial information to establish remote dependencies. Nevertheless, both the ViT and CNNs have their own limitations. For instance, a CNN is constrained by the extent of its receptive field, which prevents it from fully exploiting global spatial-spectral features. Conversely, the ViT is prone to excessive distraction during the feature extraction process. To be able to overcome the problem of insufficient feature information extraction caused using by a single paradigm, this paper proposes an MLP-mixer and a graph convolutional enhanced transformer (MGCET), whose network consists of a spatial-spectral extraction block (SSEB), an MLP-mixer, and a graph convolutional enhanced transformer (GCET). First, spatial-spectral features are extracted using SSEB, and then local spatial-spectral features are fused with global spatial-spectral features by the MLP-mixer. Finally, graph convolution is embedded in multi-head self-attention (MHSA) to mine spatial relationships and similarity between pixels, which further improves the modeling capability of the model. Correlation experiments were conducted on four different HSI datasets. The MGEET algorithm achieved overall accuracies (OAs) of 95.45%, 97.57%, 98.05%, and 98.52% on these datasets.

Compressive single-pixel spectral imaging with spatial-spectral modulation optimization via coherence minimization

Compressive spectral imaging (CSI) obtains spatial-spectral information from under-sampled measurements, which solves the contradiction between imaging speed and resolution in conventional scanning-based spectral imaging systems. The spatial-spectral modulation and the corresponding sparse prior in reconstruction algorithm jointly determine the reconstruction quality of CSI systems. In this paper, a compressive single-pixel spectral imaging system with the spatial-spectral modulation and the sparse prior simultaneously optimized by coherence minimization is proposed. By formulating the modulation of the single-pixel spectral imaging system and the sensing coherence in the differentiable matrix notation, the gradients of the modulation and the sparse prior with respect to the sensing coherence are derived for both spatial and spectral dimensions. The spatial-spectral modulation and the sparse prior are optimized via gradient descent to minimize the sensing coherence, for both the decoupled CSI systems with one dimension completely sampled and the coupled CSI system with both spatial-spectral dimensions compressively sampled. For the optimized spatial-spectral modulation, high relative mutual differences in the spectral dimension as well as low redundancy patterns in the spatial dimension are achieved, and by combining with the optimized spatial-spectral sparse prior, the enhancement of 5.7 dB for PSNR compared to the unoptimized system is realized.

Spatial-spectral encoding and dictionary optimization in compressive single-pixel hyperspectral imaging based on mutual coherence minimization.

A single-pixel detector based hyperspectral system provides an effective way to obtain the spatial-spectral information of target scenes. However, complex spectral dispersion and the substantial number of measurements not only increase the complexity of the system but also decrease the sampling efficiency and the reconstruction accuracy. In this paper, we propose a compressive sensing (CS) theory based single-pixel hyperspectral imaging system. Based on structured illumination, the spatial information is modulated by binary spatial patterns displayed on a liquid crystal on silicon (LCoS), while polarizing elements at specific angles, acting as a serious of filters, modulate the spectral dimension, effectively avoiding spectral dispersion. In terms of sampling efficiency, the application of CS significantly decreases the number of measurements required compared to the Nyquist-Shannon sampling theorem. Besides, to improve the reconstruction accuracy, mutual coherence minimization is employed to optimize the pre-trained dictionary, spatial patterns and filters. Furthermore, a two-step encoding method based on macro-pixel segmentation is proposed to address the issue of low resolution constrained by the size of the dictionary. Compared to the unoptimized system and dictionary, the proposed method achieves more accurate reconstruction results in both spectral and spatial dimensions. This work may provide opportunities for high-resolution single-pixel hyperspectral imaging systems based on CS.

PRF-Net: A Progressive Remote Sensing Image Registration and Fusion Network.

Most of the existing fusion algorithms are not robust to unregistered input images. Even after image registration, nonlinear nonregistration may persist in the local areas of the images, leading to poor quality in the fused image. So, as to tackle these challenges, a progressive remote sensing image registration and fusion network is proposed in this article, and named PRF-Net, which is particularly useful when two images are from different platforms. First, a registration network is designed to register the input image patches, which includes a global spatial transform network (GSTN) and a local spatial warp network (LSWN). The GSTN is primarily used for coarse registration, applying rigid transformation to globally align the input images. After coarse registration, the preliminarily registered moving image is input into the LSWN for local fine-tuning to maximize correlation between the input image patches. Subsequently, the fine registered images are degraded and input into the fusion network to generate the fused image. To maintain sufficient spectral and spatial information of the fused image, a multiscale feature extraction (MSFE) block with a highly interpretable spatial details attention (SDA) block is designed, which can enhance the ability of fusion network to extract and preserve spatial details and spectral information. Three groups of experiments conducted on four types of remote sensing images give evidence of that the proposed PRF-Net exhibits excellent performance in both reduced and full resolutions, showcasing its outstanding registration and fusion quality.