Spatial-spectral Fusion Research Articles

SSBFNet: a spectral-spatial fusion with BiFormer network for hyperspectral image classification

SSBFNet: a spectral-spatial fusion with BiFormer network for hyperspectral image classification

Multistage Spatial-Spectral Fusion Network for Spectral Super-Resolution.

Spectral super-resolution (SSR) aims to restore a hyperspectral image (HSI) from a single RGB image, in which deep learning has shown impressive performance. However, the majority of the existing deep-learning-based SSR methods inadequately address the modeling of spatial-spectral features in HSI. That is to say, they only sufficiently capture either the spatial correlations or the spectral self-similarity, which results in a loss of discriminative spatial-spectral features and hence limits the fidelity of the reconstructed HSI. To solve this issue, we propose a novel SSR network dubbed multistage spatial-spectral fusion network (MSFN). From the perspective of network design, we build a multistage Unet-like architecture that differentially captures the multiscale features of HSI both spatialwisely and spectralwisely. It consists of two types of the self-attention mechanism, which enables the proposed network to achieve global modeling of HSI comprehensively. From the perspective of feature alignment, we innovatively design the spatial fusion module (SpatialFM) and spectral fusion module (SpectralFM), aiming to preserve the comprehensively captured spatial correlations and spectral self-similarity. In this manner, the multiscale features can be better fused and the accuracy of reconstructed HSI can be significantly enhanced. Quantitative and qualitative experiments on the two largest SSR datasets (i.e., NTIRE2022 and NTIRE2020) demonstrate that our MSFN outperforms the state-of-the-art SSR methods. The code implementation will be uploaded at https://github.com/Matsuri247/MSFN-for-Spectral-Super-Resolution.

Tensor adaptive reconstruction cascaded with spatial-spectral fusion for marine target detection

Tensor adaptive reconstruction cascaded with spatial-spectral fusion for marine target detection

MSSFNet: A Multiscale Spatial-Spectral Fusion Network for Extracting Offshore Floating Raft Aquaculture Areas in Multispectral Remote Sensing Images.

Accurately extracting large-scale offshore floating raft aquaculture (FRA) areas is crucial for supporting scientific planning and precise aquaculture management. While remote sensing technology offers advantages such as wide coverage, rapid imaging, and multispectral capabilities for FRA monitoring, the current methods face challenges in terms of establishing spatial-spectral correlations and extracting multiscale features, thereby limiting their accuracy. To address these issues, we propose an innovative multiscale spatial-spectral fusion network (MSSFNet) designed specifically for extracting offshore FRA areas from multispectral remote sensing imagery. MSSFNet effectively integrates spectral and spatial information through a spatial-spectral feature extraction block (SSFEB), significantly enhancing the accuracy of FRA area identification. Additionally, a multiscale spatial attention block (MSAB) captures contextual information across different scales, improving the ability to detect FRA areas of varying sizes and shapes while minimizing edge artifacts. We created the CHN-YE7-FRA dataset using Sentinel-2 multispectral remote sensing imagery and conducted extensive evaluations. The results showed that MSSFNet achieved impressive metrics: an F1 score of 90.76%, an intersection over union (IoU) of 83.08%, and a kappa coefficient of 89.75%, surpassing those of state-of-the-art methods. The ablation results confirmed that the SSFEB and MSAB modules effectively enhanced the FRA extraction accuracy. Furthermore, the successful practical applications of MSSFNet validated its generalizability and robustness across diverse marine environments. These findings highlight the performance of MSSFNet in both experimental and real-world scenarios, providing reliable, precise FRA area monitoring. This capability provides crucial data for scientific planning and environmental protection purposes in coastal aquaculture zones.

A novel deep network with triangular-star spatial–spectral fusion encoding and entropy-aware double decoding for coronary artery segmentation

Coronary artery segmentation is a crucial prerequisite for computer-aided diagnosis of coronary artery disease (CAD). However, this task remains challenging due to the intricate anatomical structures and morphologies of coronary arteries (CAs), which are characterized by tortuous and numerous slender branches, large inter-subject variations, and low contrast with adjacent tissues. To address these challenges, we propose a novel deep network with triangular-star spatial–spectral fusion encoding and an entropy-aware double decoding process to comprehensively explore CA features from diverse perspectives. Specifically, to enhance the encoder’s ability to exploit spectral characteristics, we incorporate a tri-stage attention-mediated Fourier (tri-AMF) structure. This structure dynamically modulates the global features of blood vessels in the frequency domain with superior resilience against spectral noises. Simultaneously, we introduce a triangular-star cross-domain feature fusion (▽-Star fusion) module, integrating features from a pair of closely intertwined encoders dedicated to spatial and spectral domains, along with features transmitted to the decoder. This module, facilitated by richly connected pairwise interaction pathways, is designed to learn to segment coronary arteries through cross-domain deep analysis. Furthermore, our network’s decoder employs a novel local entropy-aware double decoding (LEAD2) process to adaptively fuse feature maps across all scales with the local entropy associated with each scale, explicitly modeling the network’s derivation of the final segmentation outcome. Extensive experiments on two in-house and three publicly available datasets consistently demonstrate that the proposed method has superior performance and generalization ability, outperforming multiple state-of-the-art algorithms on various metrics. The code is available at https://github.com/Cassie-CV/CASeg.

Hyperspectral object classification using hybrid spectral-spatial fusion and noise tolerant soft-margin technique

Because of its spectral-spatial and temporal resolution of greater areas, hyperspectral imaging (HSI) has found widespread application in the field of object classification. The HSI is typically used to accurately determine an object's physical characteristics as well as to locate related objects with appropriate spectral fingerprints. As a result, the HSI has been extensively applied to object identification in several fields, including surveillance, agricultural monitoring, environmental research, and precision agriculture. However, because of their enormous size, objects require a lot of time to classify; for this reason, both spectral and spatial feature fusion have been completed. The existing classification strategy leads to increased misclassification, and the feature fusion method is unable to preserve semantic object inherent features; This study addresses the research difficulties by introducing a hybrid spectral-spatial fusion (HSSF) technique to minimize feature size while maintaining object intrinsic qualities; Lastly, a soft-margins kernel is proposed for multi-layer deep support vector machine (MLDSVM) to reduce misclassification. The standard Indian pines dataset is used for the experiment, and the outcome demonstrates that the HSSF-MLDSVM model performs substantially better in terms of accuracy and Kappa coefficient.

Graph-infused hybrid vision transformer: Advancing GeoAI for enhanced land cover classification

Hyperspectral Image Classification (HSIC) is a challenging task due to the high-dimensional nature of Hyperspectral Imaging (HSI) data and the complex relationships between spectral and spatial information. This paper proposes a Graph-Infused Hybrid spatial–spectral Transformer (GFormer) for HSIC. The GFormer combines the power of graph and spatial–spectral transformer to capture both spectral relationships and spatial context. We represent the HSI data as a graph, where nodes represent pixels and edges capture spectral similarities. By incorporating an attention mechanism, the GFormer learns spatial–spectral fusion representations, allowing it to effectively discriminate between different classes. The model can capture long-range dependencies among spectral bands, enabling it to understand complex interactions in the HSI data. Moreover, the GFormer adapts to different spectral resolutions by dynamically adjusting attention weights for each spectral band. Experimental results on benchmark HSI datasets demonstrate that the GFormer outperforms state-of-the-art (SOTA) methods, achieving superior classification accuracy.

SSTF-Unet: Spatial-Spectral Transformer-Based U-Net for High-Resolution Hyperspectral Image Acquisition.

To obtain a high-resolution hyperspectral image (HR-HSI), fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) is a prominent approach. Numerous approaches based on convolutional neural networks (CNNs) have been presented for hyperspectral image (HSI) and multispectral image (MSI) fusion. Nevertheless, these CNN-based methods may ignore the global relevant features from the input image due to the geometric limitations of convolutional kernels. To obtain more accurate fusion results, we provide a spatial-spectral transformer-based U-net (SSTF-Unet). Our SSTF-Unet can capture the association between distant features and explore the intrinsic information of images. More specifically, we use the spatial transformer block (SATB) and spectral transformer block (SETB) to calculate the spatial and spectral self-attention, respectively. Then, SATB and SETB are connected in parallel to form the spatial-spectral fusion block (SSFB). Inspired by the U-net architecture, we build up our SSTF-Unet through stacking several SSFBs for multiscale spatial-spectral feature fusion. Experimental results on public HSI datasets demonstrate that the designed SSTF-Unet achieves better performance than other existing HSI and MSI fusion approaches.

A hybrid spectral-spatial fusion technique for hyperspectral object classification

<span>In the field of object classification, hyperspectral imaging (HSI) has been widely used, due to its spectral-spatial, and temporal resolution of larger areas. The HSI is generally used to identify the objects physical properties in accurate manner and as well as to identify similar object with acceptable spectral signatures. Thus, the HSI has been widely used for object identification applications in different fields such as precision agriculture, environmental study, crop monitoring, and surveillance. However, the object classification is time consuming due to extremely large size; thus, the feature fusion of both spectral and spatial have been done. The current feature fusion method fails to retain semantic object intrinsic feature; further, current classification technique induces higher misclassification. In addressing the research issues this paper introduces a hybrid spectral-spatial fusion (HSSF) technique to reduce feature size and retains object intrinsic properties. Finally, in reducing misclassification a soft-margins kernel is introduced in support vector machine (SVM). Experiment is conducted on standard Indian Pines dataset; the result shows the HSSF-SVM model attain much higher accuracy and Kappa coefficient performance.</span>

Multiscale dilated dense network for hyperspectral image classification

Hyperspectral image has rich spectral information and can accurately reflect the real condition of ground objects, so it is widely used. However, there are huge challenges in hyperspectral image, such as few training samples, the phenomena of “the foreign matter same spectrum” and “the same thingdifferent spectrum” can not be distinguished. The hyperspectral image classification method based on convolutional neural network can solve the above problems and achieve good classification performance. Most classification methods based on convolutional neural network only extract the deep semantic information at the end of the network, ignoring the shallow details. At the same time, Methods of this type increase the receptive field by designing deep network structure, which is prone to gradient vanishing. To solve the above two problems, a multiscale dilated dense network is proposed for hyperspectral image classification. Firstly, hyperspectral data cubes with different neighborhood sizes are selected as network inputs to construct a multi-channel network structure with different scale inputs. Secondly, a multiscale feature mapping module with fusion of dilated convolution is designed. Different numbers of dilated convolution are used to increase the model's receptive field without increasing network parameters, and spatial-spectral features of different scales are fused. Skip connections are added between the designed modules to fully combine the shallow and deep features of the network, without skip connections in the modules, reduce the computation of the overall network. Finally, the resulting features are input into the fully connected layer and softmax to complete the classification. In addition, in order to prevent the network from overfitting, we add dropout regularization method after the fully connected layer to improve the generalization ability of the model. Experiments are conducted on three publicly available datasets, Indian Pines, University of Pavia and Salinas, with accuracy rates of 99.28%, 99.48% and 99.25%, respectively. The experimental results show that compared with the classification methods based on convolutional neural network, the proposed method can effectively extract multiscale features of hyperspectral image and enhance the expression ability of spatial-spectral fusion features, and the classification performance is better.

Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion for remote sensing image

ABSTRACT Remote sensing image (RSI) with concurrently high spatial, temporal, and spectral resolutions cannot be produced by a single sensor. Multisource RSI fusion is a convenient technique to realize high spatial resolution multispectral (MS) images (spatial spectral fusion, i.e. SSF) and high temporal and spatial resolution MS images (spatiotemporal fusion, i.e. STF). Currently, deep learning-based fusion models can only implement SSF or STF, lacking models that perform both SSF and STF. Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion (BAPGF) for RSI are proposed to implement both SSF and STF, namely BPF-MGAN. A bidirectional adaptive-stage feature extraction architecture in fine-scale-to-coarse-scale and coarse-scale-to-fine-scale modes is introduced. The designed BAPGF introduces a previous fusion result-oriented cross-stage-level dual-residual attention fusion strategy to enhance critical information and suppress superfluous information. Adaptive resolution U-shaped discriminators are implemented to feed multiresolution context into the generator. A generalized multitask loss function unlimited by no-reference images is developed to strengthen the model via constraints on the multiscale feature, structural, and content similarities. The BPF-MGAN model is validated on SSF datasets and STF datasets. Compared with the state-of-the-art SSF and STF models, results demonstrate the superior performance of the proposed BPF-MGAN model in both subjective and objective evaluations.

Semi-supervised feature learning for disjoint hyperspectral imagery classification

With the introduction of spatial-spectral fusion and deep learning, the classification performance of hyperspectral imagery (HSI) has been promoted greatly. For some widely used datasets, the classification accuracy almost reaches 100%. However, for hyperspectral image classification, random sampling is still the most common strategy to collect the training and test samples. Because the training and test samples are randomly selected from the same image, so they have a high correlation and the classification results are overoptimistic. Besides, random sampling is not a good choice for practical applications because we cannot always collect training and test samples from the same region. Disjoint sampling selects training and testing samples from different local regions, which will provide a more objective performance evaluation for HSI classification models. In this paper, we first show the huge classification performance difference caused by different sampling strategies with a simple experiment, then we analyze the underlying reasons from the spectral information, spatial-spectral combination, sample overlapping and spatial distance, finally, a semi-supervised feature learning method is proposed for disjoint HSI classification, in which the spatial and spectral information are exploited effectively and reasonably. The experimental results based on three HSI datasets demonstrate the effectiveness of the proposed method.

Component Substitution Model-Guided Deep Spatial–Spectral Fusion of Hyperspectral Imagery

Hyperspectral (HS) spatial-spectral fusion technology is an important means to obtain high spatial and spectral resolution data for major strategic missions such as manned spaceflight and earth observation. The existing parameter-tuning black-box fusion model lacks the guidance of mathematical theory and is not explicable, so it is difficult to apply in practice. In this letter, we design an interpretable deep network for spatial-spectral fusion tasks, called S2Fusion, which embedded the mature component substitution (CS) fusion model into a deep neural network. As a result of the careful design of the method, each module in the S2Fusion network corresponds to a specific operation in the CS fusion model, which is easily interpretable. Compared with the traditional fusion models, the S2Fusion makes it easier to ensure the implementation mechanism of spatial-spectral fusion throughout the network flow. Extensive experiments demonstrate the superiority of S2Fusion both quantitatively and visually over state-of-the-art methods.

Hyperspectral Image Classification: An Analysis Employing CNN, LSTM, Transformer, and Attention Mechanism

Hyperspectral images contain tens to hundreds of bands, implying a high spectral resolution. This high spectral resolution allows for obtaining a precise signature of structures and compounds that make up the captured scene. Among the types of processing that may be applied to Hyperspectral Images, classification using machine learning models stands out. The classification process is one of the most relevant steps for this type of image. It can extract information using spatial and spectral information and spatial-spectral fusion. Artificial Neural Network models have been gaining prominence among existing classification techniques. They can be applied to data with one, two, or three dimensions. Given the above, this work evaluates Convolutional Neural Network models with one, two, and three dimensions to identify the impact of classifying Hyperspectral Images with different types of convolution. We also expand the comparison to Recurrent Neural Network models, Attention Mechanism, and the Transformer architecture.. Furthermore, a novelty pre-processing method is proposed for the classification process to avoid generating data leaks between training, validation, and testing data. The results demonstrated that using 1 Dimension Convolutional Neural Network (1D-CNN), Long Short-Term Memory (LSTM), and Transformer architectures reduces memory consumption and sample processing time and maintain a satisfactory classification performance up to 99% accuracy on larger datasets. In addition, the Transfomer architecture can approach the 2D-CNN and 3D-CNN architectures in accuracy using only spectral information. The results also show that using two or three dimensions convolution layers improves accuracy at the cost of greater memory consumption and processing time per sample. Furthermore, the pre-processing methodology guarantees the disassociation of training and testing data.

HGF Spatial–Spectral Fusion Method for Hyperspectral Images

Quantitative studies on surface elements require satellite hyperspectral images with high spatial resolution. The identification of different surface elements requires different characteristic bands and their corresponding optimal spatial–spectral fusion methods. To address these problems, the harmonic analysis (HA), guided filtering, and Gram–Schmidt (GS) algorithms were integrated to propose a spatial–spectral fusion method called HGF. The fusion experiment and validation of the hyperspectral images of GaoFen-5 (GF-5) and ZY1-02D were conducted separately using the HGF method, and the fusion effect was evaluated in three band intervals according to the spectral response of the ground class. First, HGF was used to fuse the GF-5 and GaoFen-1 (GF-1) images, and the fusion effect was evaluated both qualitatively and quantitatively. Second, the optimal fusion method was selected for the corresponding characteristic bands of the different surface elements. Finally, the hyperspectral image obtained by ZY1-02D and multispectral image of Sentinel-2B were used for validation to improve the accuracy and efficiency of satellite hyperspectral images in quantitative studies. The results show that for further studies on soil, vegetation, and water bodies, the best fusion methods in the 390–730, 730–1400, and 1400–2260 nm intervals are the GS, HGF, and HGF algorithms, respectively. Further analysis showed that the HGF or GS methods can be selected for quantitative studies on vegetation and water bodies and that the HGF method exhibits outstanding advantages for quantitative analysis of each soil element.

Neural Contourlet Network for Monocular 360° Depth Estimation

For a monocular 360° image, depth estimation is a challenging because the distortion increases along the latitude. To perceive the distortion, existing methods devote to designing a deep and complex network architecture. In this paper, we provide a new perspective that constructs an interpretable and sparse representation for a 360° image. Considering the importance of the geometric structure in depth estimation, we utilize the contourlet transform to capture an explicit geometric cue in the spectral domain and integrate it with an implicit cue in the spatial domain. Specifically, we propose a neural contourlet network consisting of a convolutional neural network and a contourlet transform branch. In the encoder stage, we design a spatial–spectral fusion module to effectively fuse two types of cues. Contrary to the encoder, we employ the inverse contourlet transform with learned low-pass subbands and band-pass directional subbands to compose the depth in the decoder. Experiments on the three popular 360° panoramic image datasets demonstrate that the proposed approach outperforms the state-of-the-art schemes with faster convergence. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhijieshen-bjtu/Neural-Contourlet-Network-for-MODE</uri> .

Deep Pansharpening via 3D Spectral Super-Resolution Network and Discrepancy-Based Gradient Transfer

High-resolution (HR) multispectral (MS) images contain sharper detail and structure compared to the ground truth high-resolution hyperspectral (HS) images. In this paper, we propose a novel supervised learning method, which considers pansharpening as the spectral super-resolution of high-resolution multispectral images and generates high-resolution hyperspectral images. The proposed method learns the spectral mapping between high-resolution multispectral images and the ground truth high-resolution hyperspectral images. To consider the spectral correlation between bands, we build a three-dimensional (3D) convolution neural network (CNN). The network consists of three parts using an encoder–decoder framework: spatial/spectral feature extraction from high-resolution multispectral images/low-resolution (LR) hyperspectral images, feature transform, and image reconstruction to generate the results. In the image reconstruction network, we design the spatial–spectral fusion (SSF) blocks to reuse the extracted spatial and spectral features in the reconstructed feature layer. Then, we develop the discrepancy-based deep hybrid gradient (DDHG) losses with the spatial–spectral gradient (SSG) loss and deep gradient transfer (DGT) loss. The spatial–spectral gradient loss and deep gradient transfer loss are developed to preserve the spatial and spectral gradients from the ground truth high-resolution hyperspectral images and high-resolution multispectral images. To overcome the spectral and spatial discrepancy between two images, we design a spectral downsampling (SD) network and a gradient consistency estimation (GCE) network for hybrid gradient losses. In the experiments, it is seen that the proposed method outperforms the state-of-the-art methods in the subjective and objective experiments in terms of the structure and spectral preservation of high-resolution hyperspectral images.

A robust and adaptive spatial-spectral fusion model for PlanetScope and Sentinel-2 imagery

ABSTRACT Satellite sensors usually compromise between their spatial and spectral resolutions due to the limitations of data volume and signal-to-noise ratio, such as the Very High spatial Resolution (VHR) PlanetScope (PS, four or five 3-m bands) and medium spatial resolution (med-resolution) Sentinel-2 (S2, ten 10-m or 20-m bands) constellations. Concomitant with the growing demand for satellite images with ample spatial details and spectral signatures, Spatial-Spectral image Fusion (SSF) is important for blending the spatial resolution of PS and the spectral resolution of S2 to produce synthetic 3-m image products in the ten bands for different applications. However, the existing studies conducted for fusing PS and S2 data present limited spatial and spectral fidelities to original PS and S2 bands. Hence, this study presents a new SSF method named Robust and Adaptive Spatial-Spectral image Fusion Model (RASSFM) for that purpose. RASSFM improves the spatial and spectral fidelities of the results through: (1) combining the spectral mapping and the spectral correlation to obtain the high-quality spatial information sources for the S2 bands, (2) utilizing the neighborhood information considering both spatial and spectral constraints to improve the spectral fidelities of the fused bands. We conducted the SSF tests at the degraded and original spatial resolutions to comprehensively evaluate the proposed RASSFM method. Furthermore, we examined the performance of RASSFM in four study sites with highly mixed regular or desert urban, heterogeneous agricultural, and vegetation-dominated landscapes. Moreover, we compared our method with twenty representative SSF methods developed for similar purposes. The results demonstrate that RASSFM not only outperforms the other methods but has robust performance in different landscapes and comparable accuracies in different bands, thereby advancing the fusion of VHR and med-resolution images from PS and S2, respectively.

A novel global-local block spatial-spectral fusion attention model for hyperspectral image classification

ABSTRACT Deep learning brought a new method for hyperspectral image (HSI) classification, in which images are usually pre-processed by reducing their dimensions before being packaged into pieces to be input to the deep network for feature extraction. However, the learning capability of convolutional kernels of fixed dimensions is usually limited, and thus they are inclined to cause losses of feature details. In this paper, a new global-local block spatial-spectral fusion attention (GBSFA) model is proposed. An improved Inception structure is designed to extract the feature information of the global block, and the self-attention mechanism and spatial pyramid pooling (SPP) are applied to focus on the interclass edge feature information of the local block. Combined with long-short term memory (LSTM) networks, the effective information of the spectral dimension is extracted. Finally, the features extracted from the spatial dimension and the spectral dimension are conveyed in the full connection layer for classification training. Experimental results show that the classification accuracy of the proposed approach is higher than that of other comparative methods using small training sets.

Hyperspectral image classification based on octave convolution and multi-scale feature fusion

Hyperspectral Image (HSI) classification is one of the main research directions of remote sensing applications. With the high dimension, strong correlation, and a large amount of HSI data, conventional classification methods often have problems like computation complexity, low classification accuracy, and poor generalization ability while extracting remote sensing image features. Moreover, the extraction of efficient features will impact the classification results. This work proposed an HSI classification method based on octave convolution and multi-scale feature fusion. The low-level features of HSI were extracted by octave convolution, and the attention mechanism was introduced into the spatial and spectral dimensions to focus on the area of interest. Then, the spectral-spatial fusion features were extracted for the classification task. Based on 300 training samples, several experiments were carried out, and the results showed that the classification accuracy of the proposed method was 99.63%, 97.90%, and 98.69% on the widely used WHU-Hi dataset. The notable observation was that fusion of features of different scales helped improve the classification performance.