Spatial-spectral Fusion Research Articles

Hyperspectral Classification via Global-Local Hierarchical Weighting Fusion Network

The fusion of spectral–spatial features based on deep learning has become the focus of research in hyperspectral image (HSI) classification. However, previous deep frameworks based on spectral–spatial fusion usually performed feature aggregation only at the branch ends. Furthermore, only first-order statistical features are considered in the fusion process, which is not conducive to improving the discrimination of spectral–spatial features. This article proposes a global–local hierarchical weighted fusion end-to-end classification architecture. The architecture includes two subnetworks for spectral classification and spatial classification. For the spectral subnetwork, two band-grouping strategies are designed, and bidirectional long short-term memory is used to capture spectral context information from global to local perspectives. For the spatial subnetwork, a pooling strategy based on local attention is combined to construct a global–local pooling fusion module to enhance the discriminability of spatial features learned by a convolutional neural network. For the fusion stage, a hierarchical weighting fusion mechanism is developed to obtain the nonlinear relationship between both spectral and spatial features. The experimental results on four real HSI datasets and a GF-5 satellite dataset demonstrate that the method proposed is more competitive in terms of accuracy and generalization.

An Efficient Dual Spatial–Spectral Fusion Network

The spectral fidelity and the spatial richness enhancement are two primary objectives for the deep learning pan-sharpening algorithms. Consequently, the two-stream fusion architecture is utilized to focus on the individual features of panchromatic (PAN) and multi-spectral (MS) images. However, most existing two-stream networks have a high degree of extracted feature redundancy and large fusion workload. Based on these, we propose a dual spatial-spectral fusion network (DSSN) to implement the divided-objective fusion, with one stream fusing spatial information and the other stream fusing spectral information. We propose a bifurcation spatial fusion stream consisted of backbone and gradient branch. The gradient feature extracted as prior knowledge to direct the backbone to enhance the spatial information richness. Moreover, the multi-scale residual block (MRB) is designed to guarantee the sufficiency of extracted features. In MRBs, the densely connected small size and asymmetric convolution kernels are used to replace the large size ones to reduce the network complexity. In the spectral fusion stream, one-dimensional convolution is adopted to efficiently fuse the spectral features of PAN and MS in the channel dimensions. Finally, the fused spatial and spectral information optimally integrate with a certain proportion to be the final result. The ablation studies confirm that each stream of DSSN is indispensable in the spectral preservation and spatial detail recovery. Three different types of optical data are used to assess practicability and universality of DSSN. The experimental results show that the fusion quality of DSSN exceeds the commonly used pan-sharpening methods, with fast training convergence speed.

Group Shuffle and Spectral-Spatial Fusion for Hyperspectral Image Super-Resolution

Recently, super-resolution (SR) tasks for single hyperspectral images have been extensively investigated and significant progress has been made by introducing advanced deep learning-based methods. However, hyperspectral image SR is still a challenging problem because of the numerous narrow and successive spectral bands of hyperspectral images. Existing methods adopt the group reconstruction mode to avoid the unbearable computational complexity brought by the high spectral dimensionality. Nevertheless, the group data lose the spectral responses in other ranges and preserve the information redundancy caused by continuous and similar spectrograms, thus containing too little information. In this paper, we propose a novel single hyperspectral image SR method named GSSR, which pioneers the exploration of tweaking spectral band sequence to improve the reconstruction effect. Specifically, we design the group shuffle that leverages interval sampling to produce new groups for separating adjacent and extremely similar bands. In this way, each group of data has more varied spectral responses and less redundant information. After the group shuffle, the spectral-spatial feature fusion block is employed to exploit the spectral-spatial features. To compensate for the adjustment of spectral order by the group shuffle, the local spectral continuity constraint module is subsequently appended to constrain the features for ensuring the spectral continuity. Experimental results on both natural and remote sensing hyperspectral images demonstrate that the proposed method achieves the best performance compared to the state-of-the-art methods.

A geometric misregistration resistant data fusion approach for adding red-edge (RE) and short-wave infrared (SWIR) bands to high spatial resolution imagery

High spatial resolution images have been widely applied to the monitoring of land surfaces during the past decades. However, satellite and unmanned aerial vehicle (UAV) sensors (e.g., IKONOS, and QuickBird) usually only provide visible and near-infrared spectral bands with a high spatial resolution. Such limited spectral bands at a high spatial resolution are not adequate for crucial environmental and agricultural studies that require essential bands (e.g., the red-edge (RE) and the short-wave infrared (SWIR)), such as land cover classification and the estimation of plant physiological parameters. On the contrary, abundant spectral information is commonly available on sensors with a medium spatial resolution (e.g., Landsat, and Sentinel-2). Existing spatial-spectral fusion methods can leverage the unique advantages of these two types of sensors, and produce synthesized images with both spatial and spectral detail. However, most spatial-spectral methods ignore the inherent geometric registration errors when using multi-source data, inevitably introducing the errors into the fusion result. To address the limitation, this study presents a new spatial-spectral fusion method, the Misregistration-Resistant Data Fusion approach (MRDF), to add new spectral bands to the high spatial resolution images from medium resolution images. The proposed method is composed of a local-model-based similar pixel searching and a global partial least square (PLS) regression, and has the following advantages: (1) the local model utilizes neighborhood similar pixels to generate predictions robust against geometric misregistration; (2) the combined model integrates a global PLS regression to improve fusion accuracy for the SWIR bands; (3) the extended-box strategy and spatial filtering can relieve geometric errors when minimizing residuals for the SWIR bands. Satellite and airborne data from three sites were used to compare the performance of MRDF with seven other typical methods. Results showed that MRDF not only produced accurate fusion results with above-average efficiency but also improved the classification accuracy, which substantiated its great potential for various applications.

Hyperspectral Image Classification Based on SE-Res2Net and Multi-Scale Spatial Spectral Fusion Attention Mechanism

<p indent=0mm>In order to extract more discriminative features for hyperspectral image and prevent the network from degrading caused by deepening, a novel multi-scale feature extraction module SE-Res2Net based on the new dimensional residual network (Res2Net) and squeeze and exception network (SENet), and a multi-scale spectral-spatial fusion attention module is developed for hyperspectral image classification. In order to overcome the degradation problem caused by network deepening, the SE-Res2Net module uses channel grouping to extract fine-grained multi-scale features of hyperspectral images, and gets multiple receptive fields of different granularity. Then, the channel optimization module is employed to quantify the importance of the feature maps at the channel level. In order to optimize the features from spatial and spectral dimensions simultaneously, a multi-scale spectral-spatial fusion attention module is designed to mine the relationship between different spatial positions and different spectral dimensions at different scales using asymmetric convolution, which can not only reduce the computation, but also effectively extract the discriminative spectral-spatial fusion features, and further improve the accuracy of hyperspectral image classification. Comparison experiments on three public datasets, Indian Pines, University of Pavia and Grss_dfc_2013 show that the proposed method has higher overall accuracy (OA), average accuracy (AA) and Kappa coefficient compared to other state-of-the-art deep networks.

Multi-scale spatial-spectral fusion based on multi-input fusion calculation and coordinate attention for hyperspectral image classification

Recently, the deep learning method that integrates image features has gradually become a hot development trend in hyperspectral image classification. However, these studies did not fully consider the fusion of image features, and did not remove the interference to the classification process caused by the difference in the size of the objects. These factors hinder the further improvement of the classification effect. To eliminate these drawbacks, this paper proposes a more effective fusion scheme (MSF-MIF), which realizes the fusion from the perspective of location characteristics and channel characteristics through 3D convolution and spatial feature concatenation. In view of the size discrepancy of the objects to be classified, this method extracts features from several input patches of different scales and uses the novel calculation method proposed to fuse them, which minimizes the interference caused by size differences. In addition, this research also tried to quote the coordinate attention structure for the first time that combines spatial and spectral attention features to further improve the classification performance. Experimental results on three commonly used data sets prove that this framework has achieved a breakthrough in classification accuracy.

HSI Classification Based on Multimodal CNN and Shadow Enhance by DSR Spatial-Spectral Fusion

Dynamic stochastic resonance (DSR) has been proved to be able to enhance the shadow region of HSIs. However, DSR can only be used for the enhancement on the spatial or spectral dimensions respectively, which destroy the correlation and integrality of HSI. Therefore, in this paper, to maintain the 3D characteristic of HSI, the spatial and spectral enhanced HSI has been fused. Furthermore, to make full use of two-modal information of HSI, a multimodal convolutional neural net-work () has been proposed for classification. In both 2D spatial modal and 3D tensor information can be integrated to improve the classification. Firstly, the shadow areas of HSI could be enhanced by DSR from spatial and spectral dimensions to get spatial enhanced HIS and spectral enhanced HSI respectively. Secondly, the enhanced HIS can be obtained by fuzing and in different proportions. Finally, can be classified by Two scenes containing shadow areas of a real-world HSI have been used in the experiment to evaluate the performance of the proposed method. The results have shown that the proposed method has a promising prospect in the classification of HSI with shadow.

A Pixel Cluster CNN and Spectral-Spatial Fusion Algorithm for Hyperspectral Image Classification With Small-Size Training Samples

Convolutional neural networks (CNNs) can automatically learn features from the hyperspectral image (HSI) data, avoiding the difficulty of manually extracting features. However, the number of training samples for the classification of HSIs is always limited, making it difficult for CNN to obtain effective features and resulting in low classification accuracy. To solve this problem, a pixel cluster CNN and spectral-spatial fusion (SSF) algorithm for hyperspectral image classification with small-size training samples is proposed in this article. First, spatial information is extracted by the gray level co-occurrence matrix. Then, spatial information and spectral information are fused by means of bands superposition, forming spectral-spatial features. To expand the number of training samples, the pixels after SSF are combined into pixel clusters according to a certain rule. Finally, a CNN framework is utilized to extract effective features from the pixel clusters. Experiments based on three standard HSIs demonstrate that the proposed algorithm can get better performance than the conventional CNN and also outperforms other studied algorithms in the case of small training set.

Spatial-spectral fusion based on band-adaptive detail injection for GF-5 and Sentinel-2 remote sensing images

Spatial-spectral fusion based on band-adaptive detail injection for GF-5 and Sentinel-2 remote sensing images

Deep Fusion of Localized Spectral Features and Multi-scale Spatial Features for Effective Classification of Hyperspectral Images

This study presents a deep extraction of localized spectral features and multi-scale spatial features convolution (LSMSC) framework for spectral-spatial fusion based classification of hyperspectral images (HSIs). First, adjacent spectral bands are grouped based on their similarity measurements, where the whole hypercube is partitioned into several sub-cubes, each corresponding to one band group. Then, the proposed localized spectral features extraction (LSF) strategy is used to extract localized spectral features, which are extracted from each band group using the 1D convolutional neural network (CNN). Meanwhile, the proposed HiASPP strategy is employed to extract the multi-scale features from the first several principal components of each sub-cube. Finally, the extracted spectral and spatial features are concatenated for spectral-spatial fusion based classification of HSI. Experiments conducted on three publicly available datasets have demonstrated that the proposed architecture outperforms several state-of-the-art approaches.

Spatial–Spectral Fusion in Different Swath Widths by a Recurrent Expanding Residual Convolutional Neural Network

The quality of remotely sensed images is usually determined by their spatial resolution, spectral resolution, and coverage. However, due to limitations in the sensor hardware, the spectral resolution, spatial resolution, and swath width of the coverage are mutually constrained. Remote sensing image fusion aims at overcoming the different constraints of remote sensing images, to achieve the purpose of combining the useful information in the different images. However, the traditional spatial–spectral fusion approach is to use data in the same swath width that covers the same area and only considers the mutually constrained conditions between the spectral resolution and spatial resolution. To simultaneously solve the image fusion problems of the swath width, spatial resolution, and spectral resolution, this paper introduces a method with multi-scale feature extraction and residual learning with recurrent expanding. To discuss the sensitivity of convolution operation to different variables of images in different swath widths, we set the sensitivity experiments in the coverage ratio and offset position. We also performed the simulation and real experiments to verify the effectiveness of the proposed framework with the Sentinel-2 data, which simulated the different widths.

DATA AUGMENTATION APPROACHES FOR SATELLITE IMAGE SUPER-RESOLUTION

Abstract. Data augmentation is a well known technique that is frequently used in machine learning tasks to increase the number of training instances and hence decrease model over-fitting. In this paper we propose a data augmentation technique that can further boost the performance of satellite image super resolution tasks. A super-resolution convolutional neural network (SRCNN) was adopted as a state-of-the-art deep learning model to test the proposed data augmentation technique. Different augmentation techniques were studied to investigate their relative importance and accuracy gains. We categorized the augmentation methods into instance based and channel based augmentation methods. The former refers to the standard approach of creating new data instances through applying image transformations to the original images such as adding artificial noise, rotations and translations to training samples, while in the latter we fuse auxiliary channels (or custom bands) with each training instance, which helps the model learn useful representations. Fusing auxiliary derived channels to a satellite image RGB combination can be seen as a spectral-spatial fusion process as we explain later. Several experiments were carried out to evaluate the efficacy of the proposed fusion-based augmentation method compared with traditional data augmentation techniques such as rotation, flip and noisy training inputs. The reconstruction quality of the high resolution output was quantitatively evaluated using Peak-Signal-To-Noise-Ratio (PSNR) and qualitatively through visualisation of test samples before and after super-resolving.

Spatial–Spectral Fusion by Combining Deep Learning and Variational Model

In the field of spatial-spectral fusion, the model-based method and the deep learning (DL)-based method are state-of-the-art. This paper presents a fusion method that incorporates the deep neural network into the model-based method for the most common case in the spatial-spectral fusion: PAN/multispectral (MS) fusion. Specifically, we first map the gradient of the high spatial resolution panchromatic image (HR-PAN) and the low spatial resolution multispectral image (LR-MS) to the gradient of the high spatial resolution multispectral image (HR-MS) via a deep residual convolutional neural network (CNN). Then we construct a fusion framework by the LR-MS image, the gradient prior learned from the gradient network, and the ideal fused image. Finally, an iterative optimization algorithm is used to solve the fusion model. Both quantitative and visual assessments on high-quality images from various sources demonstrate that the proposed fusion method is superior to all the mainstream algorithms included in the comparison in terms of overall fusion accuracy.

Spatial–Spectral Fusion Based on Conditional Random Fields for the Fine Classification of Crops in UAV-Borne Hyperspectral Remote Sensing Imagery

The fine classification of crops is critical for food security and agricultural management. There are many different species of crops, some of which have similar spectral curves. As a result, the precise classification of crops is a difficult task. Although the classification methods that incorporate spatial information can reduce the noise and improve the classification accuracy, to a certain extent, the problem is far from solved. Therefore, in this paper, the method of spatial–spectral fusion based on conditional random fields (SSF-CRF) for the fine classification of crops in UAV-borne hyperspectral remote sensing imagery is presented. The proposed method designs suitable potential functions in a pairwise conditional random field model, fusing the spectral and spatial features to reduce the spectral variation within the homogenous regions and accurately identify the crops. The experiments on hyperspectral datasets of the cities of Hanchuan and Honghu in China showed that, compared with the traditional methods, the proposed classification method can effectively improve the classification accuracy, protect the edges and shapes of the features, and relieve excessive smoothing, while retaining detailed information. This method has important significance for the fine classification of crops in hyperspectral remote sensing imagery.

Forest Types Classification Based on Multi-Source Data Fusion

Forest plays an important role in global carbon, hydrological and atmospheric cycles and provides a wide range of valuable ecosystem services. Timely and accurate forest-type mapping is an essential topic for forest resource inventory supporting forest management, conservation biology and ecological restoration. Despite efforts and progress having been made in forest cover mapping using multi-source remotely sensed data, fine spatial, temporal and spectral resolution modeling for forest type distinction is still limited. In this paper, we proposed a novel spatial-temporal-spectral fusion framework through spatial-spectral fusion and spatial-temporal fusion. Addressing the shortcomings of the commonly-used spatial-spectral fusion model, we proposed a novel spatial-spectral fusion model called the Segmented Difference Value method (SEGDV) to generate fine spatial-spectra-resolution images by blending the China environment 1A series satellite (HJ-1A) multispectral image (Charge Coupled Device (CCD)) and Hyperspectral Imager (HSI). A Hierarchical Spatiotemporal Adaptive Fusion Model (HSTAFM) was used to conduct spatial-temporal fusion to generate the fine spatial-temporal-resolution image by blending the HJ-1A CCD and Moderate Resolution Imaging Spectroradiometer (MODIS) data. The spatial-spectral-temporal information was utilized simultaneously to distinguish various forest types. Experimental results of the classification comparison conducted in the Gan River source nature reserves showed that the proposed method could enhance spatial, temporal and spectral information effectively, and the fused dataset yielded the highest classification accuracy of 83.6% compared with the classification results derived from single Landsat-8 (69.95%), single spatial-spectral fusion (70.95%) and single spatial-temporal fusion (78.94%) images, thereby indicating that the proposed method could be valid and applicable in forest type classification.

Spectral-spatial fusion model for robust blood pulse waveform extraction in photoplethysmographic imaging.

Photoplethysmographic imaging is an optical solution for non-contact cardiovascular monitoring from a distance. This camera-based technology enables physiological monitoring in situations where contact-based devices may be problematic or infeasible, such as ambulatory, sleep, and multi-individual monitoring. However, automatically extracting the blood pulse waveform signal is challenging due to the unknown mixture of relevant (pulsatile) and irrelevant pixels in the scene. Here, we propose a signal fusion framework, FusionPPG, for extracting a blood pulse waveform signal with strong temporal fidelity from a scene without requiring anatomical priors. The extraction problem is posed as a Bayesian least squares fusion problem, and solved using a novel probabilistic pulsatility model that incorporates both physiologically derived spectral and spatial waveform priors to identify pulsatility characteristics in the scene. Evaluation was performed on a 24-participant sample with various ages (9-60 years) and body compositions (fat% 30.0 ± 7.9, muscle% 40.4 ± 5.3, BMI 25.5 ± 5.2 kg·m-2). Experimental results show stronger matching to the ground-truth blood pulse waveform signal compared to the FaceMeanPPG (p < 0.001) and DistancePPG (p < 0.001) methods. Heart rates predicted using FusionPPG correlated strongly with ground truth measurements (r2 = 0.9952). A cardiac arrhythmia was visually identified in FusionPPG's waveform via temporal analysis.

Spatial-spectral operator theoretic methods for hyperspectral image classification

With the emergence of new remote sensing modalities, it becomes increasingly important to find novel algorithms for fusion and integration of different types of data for the purpose of improving performance of applications, such as target/anomaly detection or classification. Many popular techniques that deal with this problem are based on performing multiple classifications and fusing these individual results into one product. In this paper we provide a new approach, focused on creating joint representations of the multi-modal data, which then can be subject to analysis by state of the art classifiers. In the work presented in this paper we consider the problem of spatial-spectral fusion for hyperspectral imagery. Our approach involves machine learning techniques based on analysis of joint data-dependent graphs and the resulting data-dependent fusion operators and their representations.