Multispectral Image Fusion Research Articles

KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion

In recent years, the fusion of hyperspectral and multispectral images in remote sensing image processing still faces challenges, primarily due to their complexity and multimodal characteristics. Diffusion models, known for their stable training process and exceptional image generation capabilities, have shown good application potential in this field. However, when dealing with multimodal data, it may prove challenging for the models to fully capture the intricate relationships between the modalities, which may result in incomplete information integration and a small amount of remaining noise in the generated images. To address these problems, we propose a new model, KanDiff, for hyperspectral and multispectral image fusion. To address the differences in modal information between multispectral and hyperspectral images, KANDiff incorporates Kolmogorov–Arnold Networks (KAN) to guide the inputs. It helps the model understand the complex relationships between the modalities by replacing the fixed activation function in the traditional MLP with a learnable activation function. Furthermore, the image generated by the diffusion model may exhibit a small amount of the remaining noise. Convolutional Neural Networks (CNNs) effectively extract local features through their convolutional layers and achieve noise suppression via layer-by-layer feature representation. Therefore, the MergeCNN module is further introduced to enhance the fusion effect, resulting in smoother and more accurate outcomes. Experimental results on the public CAVE and Harvard datasets indicate that KanDiff has achieved improvements over current high-performance methods across several metrics, particularly showing significant enhancements in the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM), thus demonstrating superior performance. Additionally, we have created an image fusion dataset of the lunar surface, and KANDiff exhibits robust performance on this dataset as well. This work introduces an effective solution for addressing the challenges posed by missing high-resolution hyperspectral images (HRHS) data, which is essential for tasks including landing site selection and resource exploration within the realm of deep space exploration.

Graph-Based Tensor Bayesian Multispectral Image Fusion With Nonconvex Tensor Rank Minimization Model

Graph-Based Tensor Bayesian Multispectral Image Fusion With Nonconvex Tensor Rank Minimization Model

NGST-Net: A N-Gram based Swin Transformer Network for improving multispectral and hyperspectral image fusion

ABSTRACT Transformer-based deep networks have been widely employed for fusing multispectral and hyperspectral images because the global self-attention mechanism ensures that more pixel information is used in the fusion. However, the unreasonable information interactions between local windows in window self-attention-based Transformer can adversely affect the network’s modeling of the global relationship, resulting in low-quality fusion results. This study proposes a novel N-Gram based Swin Transformer Network (NGST-Net). It utilizes more pixels for modeling the global relationship, mining more spectral information of similar pixels. A N-Gram strategy is proposed to learn the spectral feature relationship between the local window and previous and subsequent windows to guide the information interactions between the local windows. A maskless shifted window strategy enables the efficient modeling of the global relationship. Experimental results show that the proposed NGST-Net has fewer parameters, higher inference speed, and smaller memory footprint than several recently published Transformer methods. The improvements in the SAM, ERGAS, and PSNR over the second-best method are 9.9%, 6.5%, and 1%, respectively, on the CAVE dataset. The inference speed of NGST-Net is 10.7 times faster than that of Fusformer. The proposed NGST-Net is a novel deep network model for the effective fusion of multispectral and hyperspectral images.

Multi-spectral Remote Sensing Image Fusion Method Based on Gradient Moment Matching

Image fusion is a popular research direction in the field of computer vision. Traditional image fusion algorithms can achieve good results in fusing grayscale images, but it is difficult to achieve ideal results in processing multi-spectral images. To address the shortcomings of multi-spectral image fusion, this study proposes a low computational complexity and low latency multi-spectral image fusion model by utilizing a multi-step degree moment matching algorithm and a generative adversarial network for fusion. Through experiments, it was found that the F1 score of the GAN-MMN model on the TinyPerson dataset was 89.79%, with an average recall rate of 89.76%. The GAN-MMN performance was higher than that of the control model. Meanwhile, the GAN-MMN model also exhibited superior performance in high-frequency feature extraction and time delay compared to the control model. According to the experimental results, the proposed multi-spectral remote sensing image fusion model had a high feature extraction effect, and its recall rate and F1 score were better than the control model, so the research model had a certain progressiveness. The proposal of this model gives a new approach for the processing of multi-spectral remote sensing images, effectively promoting the development of the computer vision industry.

SSTF-Unet: Spatial-Spectral Transformer-Based U-Net for High-Resolution Hyperspectral Image Acquisition.

To obtain a high-resolution hyperspectral image (HR-HSI), fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) is a prominent approach. Numerous approaches based on convolutional neural networks (CNNs) have been presented for hyperspectral image (HSI) and multispectral image (MSI) fusion. Nevertheless, these CNN-based methods may ignore the global relevant features from the input image due to the geometric limitations of convolutional kernels. To obtain more accurate fusion results, we provide a spatial-spectral transformer-based U-net (SSTF-Unet). Our SSTF-Unet can capture the association between distant features and explore the intrinsic information of images. More specifically, we use the spatial transformer block (SATB) and spectral transformer block (SETB) to calculate the spatial and spectral self-attention, respectively. Then, SATB and SETB are connected in parallel to form the spatial-spectral fusion block (SSFB). Inspired by the U-net architecture, we build up our SSTF-Unet through stacking several SSFBs for multiscale spatial-spectral feature fusion. Experimental results on public HSI datasets demonstrate that the designed SSTF-Unet achieves better performance than other existing HSI and MSI fusion approaches.

Progressive fusion of hyperspectral and multispectral images based on joint bilateral filtering

Progressive fusion of hyperspectral and multispectral images based on joint bilateral filtering

Self-supervised spectral super-resolution for a fast hyperspectral and multispectral image fusion

Hyperspectral-multispectral image fusion (HSI-MSI Fusion) for enhancing resolution of hyperspectral images is a hot topic in remote sensing. An important category of approaches for HSI-MSI Fusion is based on deep learning. The main challenges in deep learning based fusion methods include the lack of training data, poor generalization to various datasets, and high computational costs. This paper suggests a new approach to tackle these difficulties by introducing an innovative technique for HSI-MSI fusion. The proposed method involves training a tiny deep neural network that can reconstruct high-resolution hyperspectral images through spectral super-resolution of high-resolution multispectral images. This method does not require high resolution training data and they are artificially generated based on the spatial degradation model of the input observation images. Therefore, the problems of data scarcity and poor generalization are addressed, and also the computational burden is significantly reduced. After conducting thorough experiments, it was found that the proposed method provides promising results. The source code of this method is available at https://github.com/rajaei-arash/SSSR-HSI-MSI-Fusion.

A method based on hybrid cross-multiscale spectral-spatial transformer network for hyperspectral and multispectral image fusion

Convolutional neural networks (CNNs) have made a significant contribution to hyperspectral image (HSI) generation. However, capturing long-range dependencies can be challenging with CNNs due to the limitations of their local receptive fields, which can lead to distortions in fused images. Transformers excel at capturing long-range dependencies but have limited capacity for handling fine details. Additionally, priorwork has often overlooked the extraction of global features during the image preprocessing stage, resulting in the potential loss of fine details. To address these issues, we propose a hybrid cross-multiscale spectral-spatial Transformer (HCMSST) that combines the advantages of CNNs in feature extraction and Transformers in capturing long-range dependencies. To fully extract and retain local and global information in the shallow feature extraction phase, the network incorporatesCNNs with a staggered cascade-dense residual block (SCDRB). This block employs staggered residuals to establish direct connections bothwithin and between branches and integrates attention modules to enhance the response to important features. This approach facilitates unrestricted information exchange and fosters deeper feature representations. To address the limitationsof Transformer in processing fine details, we introduce multiscale spatial-spectral coding-decoding structures to obtain comprehensive spatial-spectral features, which are utilized to capture the long-range dependencies via the cross-multiscale spectral-spatial Transformer (CMSST). Further, the CMSST incorporates a cross-level dual-stream feature interaction strategy that integrates spatial and spectral features from different levels and then feeds the fused features back to their corresponding branches for information interaction. Experimental results indicate that the proposed HCMSST achieves superior performance compared to many state-of-the-art (SOTA) methods. Specifically, HCMSST reduces the ERGAS metric by 3.05% compared to the SOTA methods on the CAVE dataset, while on the Harvard dataset, it achieves a 2.69% reduction in ERGAS compared to the SOTA results.

Hyperspectral and multispectral images fusion based on pyramid swin transformer

Remote sensing image fusion aims to generate a high spatial resolution hyperspectral image (HR-HSI) by integrating a low spatial resolution hyperspectral image (LR-HSI) and a high spatial resolution multispectral image (HR-MSI). While Convolutional Neural Networks (CNNs) have been widely employed in addressing the HSI-MSI fusion problem, their limited receptive field poses challenges in capturing global relationships within the feature maps. On the other hand, the computational complexity of Transformers hinders their application, especially in dealing with high-dimensional data like hyperspectral images (HSIs). To overcome this challenge, we propose an HSI-MSI fusion method based on the Pyramid Swin Transformer (PSTF). The pyramid design of the PSTF effectively extracts multi-scale information from images. The Spatial–Spectral Crossed Attention (SSCA) module, comprising the Cross Spatial Attention (CSA) and the Spectral Feature Integration (SFI) modules. The CSA module employs a cross-shaped self-attention mechanism, providing greater modeling flexibility for different spatial scales and non-local structures compared to traditional convolutional layers. Meanwhile, the SFI module introduces a global memory block (MB) to select the most relevant low-rank spectral vectors, integrating global spectral information with local spatial–spectral correlation to better extract and preserve spectral information. Additionally, the Separate Feature Extraction (SFE) module enhances the network’s ability to represent image features by independently processing positive and negative parts of shallow features, thus capturing details and structures more effectively and preventing the vanishing gradient problem. Compared with the state-of-the-art (SOTA) methods, experimental results demonstrate the effectiveness of the PSTF method.

Spectral-Spatial Transformer for Hyperspectral Image Sharpening.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Multispectral and hyperspectral image fusion (MS/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods.

Multispectral and hyperspectral image fusion via multi-model deep prior regularization

ABSTRACT Due to the difficulty of collecting a sufficient number of photons within a narrow spectral window, hyperspectral images often have low spatial resolution. In order to solve this problem without increasing hardware costs, a large number of image super-resolution algorithms have been proposed and have shown promising results. However, the performance of these algorithms is inconsistent in different datasets. To fix this situation, we proposed a multi-model deep prior regularization method for multispectral and hyperspectral image fusion. This method leverages mechanistic knowledge to combine various super-resolution results into a single algorithm, significantly improving the quality of the hyperspectral images. Specifically, we first proposed a variable residual mixed attention network. This network takes a low-resolution hyperspectral image and a high-resolution multispectral image as inputs, producing deep prior results from three different modes by adjusting spatial and channel attention. Then, we reconstructed the prior results along the spectral dimension and used them as regularization terms in our mechanistic model. We transformed the problem of finding the optimal solution of the model into solving the Sylvester equation and applied a fast solver for this equation to create the high-resolution hyperspectral images. Additionally, we used a genetic algorithm to determine the best hyperparameters for different deep prior results. Through validation on three datasets, we have demonstrated that our method performs better than existing methods in both quantitative and qualitative comparisons.

Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network With Spatial-Spectral Manifold Learning.

Hyperspectral image (HSI) and multispectral image (MSI) fusion aims to generate high spectral and spatial resolution hyperspectral image (HR-HSI) by fusing high-resolution multispectral image (HR-MSI) and low-resolution hyperspectral image (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, and incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for LR-HSI and HR-MSI based on Tucker decomposition and spatial-spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network (CTFN) that incorporates a spatial-spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity to capture global information, a Laplacian-based spatial-spectral manifold constraint is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.

Advancing Skarn Iron Ore Detection through Multispectral Image Fusion and 3D Convolutional Neural Networks (3D-CNNs)

This study explores novel techniques to improve the detection accuracy of skarn iron deposits using advanced image-processing methodologies. Leveraging the capabilities of ASTER image, band ratio (BR) images, and principal component analysis (PCA) alongside the power of 3D convolutional neural networks (3D-CNNs), the research aims to enhance the precision and efficiency of ore detection in complex geological environments. The proposed method employs a specific 3D-CNN architecture accepting input as a 7 × 7 × C image patch, where C represents the combined number of selected ASTER image bands, principal component (PC) bands, and computed BR images. To evaluate the accuracy of the proposed method, five distinct image band combinations, including the proposed band combination, were tested and evaluated based on the overall accuracy (OA), average accuracy (AA), and kappa coefficient. The results demonstrated that while the incorporation of BR images alongside ASTER bands initially seemed promising, it introduced significant confusion in certain classifications, leading to unexpected misclassification rates. Surprisingly, utilizing solely ASTER bands as input parameters yielded higher accuracy rates (OA = 93.13%, AA = 91.96%, kappa = 90.91%) compared with scenarios involving the integration with band ratios (OA = 87.02%, AA = 79.15, kappa = 82.60%) or the integration of BR images to PC bands (OA = 87.78%, AA = 82.39%, kappa = 83.81%). However, the amalgamation of ASTER bands with selected PC bands showed slight improvements in accuracy (OA = 94.65%, AA = 92.93%, kappa = 93.45%), although challenges in accurately classifying certain features persisted. Ultimately, the proposed combination of ASTER bands, PC bands, and BR images (proposed band combination) presented the most visually appealing and statistically accurate results (OA = 96.95%, AA = 94.87%, kappa = 95.93%), effectively addressing misclassifications observed in the other combinations. These findings underscore the synergistic contributions of each of the ASTER bands, PC bands, and BR images, with the ASTER bands proving pivotal for optimal skarn classification, the PC bands enhancing intrusions classification accuracy, and the BR images strengthening wall rock classification accuracy. In conclusion, the proposed combination of input image bands emerges as a robust and comprehensive methodology, demonstrating unparalleled accuracy in the remote sensing detection of skarn iron minerals.

Multi-Stage Feature Fusion of Multispectral and SAR Satellite Images for Seasonal Crop-Type Mapping at Regional Scale Using an Adapted 3D U-Net Model

Multi-Stage Feature Fusion of Multispectral and SAR Satellite Images for Seasonal Crop-Type Mapping at Regional Scale Using an Adapted 3D U-Net Model

Earth Observation Multi-Spectral Image Fusion with Transformers for Sentinel-2 and Sentinel-3 Using Synthetic Training Data

Earth Observation Multi-Spectral Image Fusion with Transformers for Sentinel-2 and Sentinel-3 Using Synthetic Training Data

Multi-spectral image fusion for moving object detection

This study delves into the problem of moving object detection in infrared and visible images. While existing approaches primarily focus on single-task detection using single-spectral image data, such as thermal infrared or visible images, they often ignore the differences between different spectral images and the interconnectedness of related tasks such as image fusion and segmentation. To tackle these problems, we present a novel multi-spectral image fusion network with quality and semantic awareness for moving object detection (MOD), particularly in scenarios where ground truth labels for both infrared and visible images are unavailable. Our approach fuses multi-spectral images, incorporating additional subtasks in the image fusion to obtain content and quality perception of infrared and visible images. In addition, we design a novel residual global perception module (RGPM) and multi-spectral fusion loss, which can capture more hidden features and contextual information across various scales. This enhanced capability leads to more precise detection and tracking of moving objects, particularly in challenging situations involving occlusions and dynamic backgrounds. Compared with single-spectral moving object detection optimization, the hurdles of utilizing deep learning for multi-spectral image fusion, e.g., without ground truth labels and harmful noise, are significantly mitigated. Extensive quantitative and qualitative comparative experiments demonstrate its effectiveness, robustness, and superior performance compared to contemporary methods. Concisely, the proposed fusion representation learning has gained 44.2%,31.2%,36.3%,41.6%,5.3%,31.2%,76.2% on EI, SF, DF, AG, MI, SD and Nabf metrics compared with the best competitors.

ReLAP-Net: Residual Learning and Attention Based Parallel Network for Hyperspectral and Multispectral Image Fusion

Remote sensing applications require high-resolution images to obtain precise information about the Earth???s surface. Multispectral images have high spatial resolution but low spectral resolution. Hyperspectral images have high spectral resolution but low spatial resolution. This study proposes a residual learning and attention-based parallel network based on residual network and channel attention. The network performs image fusion of a high spatial resolution multispectral image and a low spatial resolution hyperspectral image. The network training and fusion experiments are conducted on four public benchmark data sets to show the effectiveness of the proposed model. The fusion performance is compared with classical signal processing???based image fusion techniques. Four image metrics are used for the quantitative evaluation of the fused images. The proposed network improved fusion ability by reducing the root mean square error and relative dimensionless global error in synthesis and increased the peak signal-to-noise ratio when compared to other state-of-the-art models.

UPGAN: An Unsupervised Generative Adversarial Network Based on U-Shaped Structure for Pansharpening

Pansharpening is the fusion of panchromatic images and multispectral images to obtain images with high spatial resolution and high spectral resolution, which have a wide range of applications. At present, methods based on deep learning can fit the nonlinear features of images and achieve excellent image quality; however, the images generated with supervised learning approaches lack real-world applicability. Therefore, in this study, we propose an unsupervised pansharpening method based on a generative adversarial network. Considering the fine tubular structures in remote sensing images, a dense connection attention module is designed based on dynamic snake convolution to recover the details of spatial information. In the stage of image fusion, the fusion of features in groups is applied through the cross-scale attention fusion module. Moreover, skip layers are implemented at different scales to integrate significant information, thus improving the objective index values and visual appearance. The loss function contains four constraints, allowing the model to be effectively trained without reference images. The experimental results demonstrate that the proposed method outperforms other widely accepted state-of-the-art methods on the QuickBird and WorldView2 data sets.

Balanced spatio-spectral feature extraction for hyperspectral and multispectral image fusion

Although remote sensing sensors acquire hyperspectral images (HSI) with outstanding spectral resolution, hardware limitations prevent them from obtaining HSI with superior spatial resolution. Having high-resolution multispectral images (MSI) from the same scene has sparked the concept of HSI-MSI fusion to achieve high-resolution HSI. Recently, deep learning algorithms have shown excellent performance in HSI-MSI fusion. Deep learning algorithms can automatically extract the priors latent in the input data and reconstruct the output image. However, their performance depends on the training data’s inherent characteristics. HSI volume is generally significantly larger than MSI volume. While the high-resolution MSI provides richer spatial information, its contribution to fusion is lower due to its smaller volume. To address this volume discrepancy, a deep convolutional neural network named BASFE is proposed. This network employs a two-branch structure that balances the extracted spatio-spectral features from both HSI and MSI input images, enabling effective feature fusion. Therefore, better efficiency can be achieved by using simpler structures. Experiments showed that balancing the data improves the model’s performance significantly. Furthermore, to improve efficiency and reduce complexity and computational burden, the spatio-spectral feature extraction residual modules are embedded in a dense architecture to jointly extract these features. Comprehensive experiments on five challenging datasets show that BASFE surpasses state-of-the-art algorithms. Compared to the first- and second-best competing methods, the proposed method achieves an average improvement of 1.58 dB and 3.68 dB in peak signal-to-noise ratio (PSNR) across the five studied datasets. The source code of BASFE is available in https://github.com/rajaei-arash/BASFE_hyperspectral_fusion.git.