Remote Sensing Image Pansharpening Research Articles

A novel cross fusion model with fine-grained detail reconstruction for remote sensing image pan-sharpening

ABSTRACT Pan-sharpening aims to obtain high resolution multispectral (HRMS) images by integrating the information in the panchromatic and multispectral images. Existing pan-sharpening methods have demonstrated impressive sharpening performance. However, these methods inherently overlook the complementary characteristics and interaction between diverse source images, resulting in sharpened outcomes accompanied by distortion. To solve the above problems, we construct a novel cross fusion model with fine-grained detail reconstruction from the perspective of frequency-domain. The motivation of the model is twofold: (1) to reconstruct spatial detail representations from diverse source images, laying the foundation for the generation of fine details in the subsequent fused images; and (2) to enhance the interaction between diverse source features during the fusion process in order to attain high-fidelity fusion outcomes. Based on the theoretical model, we develop a frequency-spectral dual domain cross fusion network (CF2N) utilizing the deep learning technique. Consequently, the CF2N consist of two main stages, namely frequency-domain dominated detail reconstruction (FD2R) and frequency-spectral cross fusion (FSCF). Specifically, a more reasonable reconstruction of fine frequency details in HRMS can be achieved by performing adaptive weighted fusion of frequency details in the FD2R stage. Furthermore, the FSCF module, which seamlessly integrates frequency- and spectral-domain details in a highly interactive cross fusion manner. As a result, the CF2N possesses the capability to attain high frequency-spectral fidelity results with excellent interpretability. Extensive experiments show the superior performance of ours over state of the art, while maintaining high efficiency. All implementations of this work will be published at our website.

Remote Sensing Image Pansharpening Using Deep Internal Learning With Residual Double-Attention Network

Remote Sensing Image Pansharpening Using Deep Internal Learning With Residual Double-Attention Network

A Swin Transformer with Dynamic High-Pass Preservation for Remote Sensing Image Pansharpening

Pansharpening is a technique used in remote sensing to combine high-resolution panchromatic (PAN) images with lower resolution multispectral (MS) images to generate high-resolution multispectral images while preserving spectral characteristics. Recently, convolutional neural networks (CNNs) have been the mainstream in pansharpening by extracting the deep features of PAN and MS images and fusing these abstract features to reconstruct high-resolution details. However, they are limited by the short-range contextual dependencies of convolution operations. Although transformer models can alleviate this problem, they still suffer from weak capability in reconstructing high-resolution detailed information from global representations. To this end, a novel Swin-transformer-based pansharpening model named SwinPAN is proposed. Specifically, a detail reconstruction network (DRNet) is developed in an image difference and residual learning framework to reconstruct the high-resolution detailed information from the original images. DRNet is developed based on the Swin Transformer with a dynamic high-pass preservation module with adaptive convolution kernels. The experimental results on three remote sensing datasets with different sensors demonstrate that the proposed approach performs better than state-of-the-art networks through qualitative and quantitative analysis. Specifically, the generated pansharpening results contain finer spatial details and richer spectral information than other methods.

Retracted: Artificial Intelligence-Based Deep Fusion Model for Pan-Sharpening of Remote Sensing Images.

[This retracts the article DOI: 10.1155/2021/7615106.].

Mun-GAN: A Multiscale Unsupervised Network for Remote Sensing Image Pansharpening

In remote sensing image fusion, pansharpening is a type of remote sensing image fusion method that aims to fuse panchromatic (PAN) images and multispectral (MS) images to produce high-resolution multispectral (HRMS) images. Deep learning based pansharpening technology offers a series of advanced unsupervised algorithms. However, there are several challenges: (1) The existing unsupervised pansharpening methods only consider the fusion of single-scale features; (2) for the fusion of MS and PAN image feature branches, the existing pansharpening methods are implemented directly by concatenation and summation, without paying attention to critical features or suppressing redundant features; (3) the semantic gap in the long skip connections of the network architecture will create unexpected results. In this paper, we design a multiscale unsupervised architecture based on generative adversarial networks (GANs) for remote sensing image pansharpening (Mun-GAN), which consists of a generator and two discriminators. The generator includes a multi-scale feature extractor (MFE), a self-adaptation weighted fusion (SWF) module, and a nest feature aggregation (NFA) module. First, the MFE is utilized to extract multiscale feature information from the input images and to then pass this information to the SWF module for adaptive weight fusion. Then, multiscale features are reconstructed by the NFA module to obtain HRMS images. The two discriminators are spectral and spatial discriminators used against the generator. Moreover, we design a hybrid loss function to aggregate the multiscale spectral and spatial feature information. Compared with other state-of-the-art methods using QuickBird, GaoFen-2 and WorldView-3 images, which demonstrate that the Mun-GAN yields better fusion results.

DiTBN: Detail Injection-Based Two-Branch Network for Pansharpening of Remote Sensing Images

Pansharpening is one of the main research topics in the field of remote sensing image processing. In pansharpening, the spectral information from a low spatial resolution multispectral (LRMS) image and the spatial information from a high spatial resolution panchromatic (PAN) image are integrated to obtain a high spatial resolution multispectral (HRMS) image. As a prerequisite for the application of LRMS and PAN images, pansharpening has received extensive attention from researchers, and many pansharpening methods based on convolutional neural networks (CNN) have been proposed. However, most CNN-based methods regard pansharpening as a super-resolution reconstruction problem, which may not make full use of the feature information in two types of source images. Inspired by the PanNet model, this paper proposes a detail injection-based two-branch network (DiTBN) for pansharpening. In order to obtain the most abundant spatial detail features, a two-branch network is designed to extract features from the high-frequency component of the PAN image and the multispectral image. Moreover, the feature information provided by source images is reused in the network to further improve information utilization. In order to avoid the training difficulty for a real dataset, a new loss function is introduced to enhance the spectral and spatial consistency between the fused HRMS image and the input images. Experiments on different datasets show that the proposed method achieves excellent performance in both qualitative and quantitative evaluations as compared with several advanced pansharpening methods.

Attention-Based and Staged Iterative Networks for Pansharpening of Remote Sensing Images

The pansharpening method combines complementary features from panchromatic (PAN) images and multispectral (MS) images to provide high-resolution MS images. Therefore, how to extract the features completely and reconstruct the image with high quality is the key link to obtain the ideal fusion image. We propose an attention-based and staged iterative network (ASIN) framework, which considers each subnetwork of the iterative network as a multistage process of pansharpening and carries out feature extraction and image reconstruction in each stage. Using the advantages of an iterative network for cross-stage depth features for the hierarchical extraction of refined features of MS images and PAN images for image reconstruction, we use the large kernel attention (LKA) module and the cascaded asymmetric coupling representation module to build the framework for feature extraction and the attention fusion module (AFM) to fuse the features of PAN and MS in the image reconstruction stage. LKA has channel and spatial adaptability, as well as strong long-range dependency establishment capability, which makes the feature extraction more complete. Asymmetric coupled representation module (ACRM) outputs refined spectral and spatial features by learning the hybrid correlation of MS and PAN images. AFM effectively utilizes the spectral and spatial features of the input, enabling the network to reduce information loss and retain important information. On the QuickBird (QB), WorldView-2 (WV2), and Gaofen-2 (GF-2) datasets, the superior performance of our method over the contrasting methods is demonstrated by quantitative comparison and qualitative analysis.

Self-Supervised Pansharpening Based on a Cycle-Consistent Generative Adversarial Network

In the field of remote sensing image pansharpening, deep learning-based methods have shown impressive performances recently. However, most deep learning-based pansharpening methods are based on supervised learning, which requires a large number of training images. In addition, obtaining large amounts of images with a high spatial and spectral resolution for training may be difficult in practice. In this letter, a novel self-supervised learning method based on a cycle-consistent generative adversarial network (CycleGAN) is proposed for remote sensing image pansharpening, without requiring large volumes of data for training. The framework contains two generators and two discriminators, and applies a residual neural network to the first generator. The panchromatic (PAN) image and multispectral (MS) image are input into the first generator to obtain the fused image, and then the fused image is input into the second generator to obtain a PAN image, which should be consistent with the input PAN image. The experimental results show that the proposed method performs better than the state-of-the-art unsupervised pansharpening method, and also achieves a competitive performance when compared with a supervised method.

DADR: Dual Attention Based Dual Regression Network for Remote Sensing Image Pan-Sharpening

DADR: Dual Attention Based Dual Regression Network for Remote Sensing Image Pan-Sharpening

A Knowledge Optimization-Driven Network With Normalizer-Free Group ResNet Prior for Remote Sensing Image Pan-Sharpening

Multispectral images play a crucial role in environmental monitoring or ecological analysis for their large scope, quick acquisition, and big data. With the rapid development of technology and increasing demand, very high-resolution multispectral images have attracted a lot of attention these days. However, due to sensor equipment and the imaging environment, the spatial resolution of multispectral images is always restricted. With the help of panchromatic images, pan-sharpening is a very important technique to enhance the spatial details of multispectral images. In this study, we proposed a knowledge optimization-driven pan-sharpening network with normalizer-free group ResNet prior, called PNXnet, which is unfolded from a physical knowledge optimization-driven variational model. We solved the memory overhead brought by the traditional ResNet relying on batch normalization. Results on four sensors show that high quantitative indexes and natural visual effects have verified the reliability of PNXnet. Focusing on the NIR band where spatial details are hard to be injected, we compared the Normalized Difference Vegetation Index (NDVI) generated from the fused results, the estimated NDVI shows a high consistency to the ground truth with R² above 0.91. Besides, we also compared the model generation. Furthermore, low model complexity and quicker computational speed make the daily application of PNXnet possible.

Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks

The goal of pansharpening technique is to reconstruct a high resolution spectral image (HRMS) from a low spatial resolution multispectral image (MS) and a high spatial resolution single band panchromatic (PAN) image. As the efficacy of such a technique relies upon its capability to reinforce the spatial information of the MS image while conserving its spectral signature, in this paper, we propose DI-GAN, an effective Detail Injection Generative Adversarial Network for remote sensing image pansharpening. DI-GAN employs a deep two-stream Convolutional Neural Network (CNN) generator model to combine the spatial information available on the PAN image and the spectral characteristic belonging to the MS image, in order to predict the high-frequency detail to be accurately injected into the MS image to reconstruct the desired HRMS. An original loss function is particularly proposed to encourage the network to predict only high-frequency detail. DI-GAN incorporates in parallel a Relativistic Discriminator that allows the pansharpening products to be more realistic. Experimental results at degraded and full scale on three different datasets were showed that the proposed fusion approach can produce superior pansharpening performances in terms of spatial and spectral fidelities with respect to the literature techniques, including CNN- and GAN-based methods.

Remote sensing image fusion with multistream deep ResCNN

Remote sensing image pan-sharpening is the fusion of multispectral (MS) image and panchromatic (PAN) image into fused image with MS and high-resolution information features. In recent years, deep learning technology has been widely used in abounding fields such as text and image recognition. Inspired by this, we proposed a multistream deep ResCNN (RIFRCNN)-based fusion method for remote sensing images. First, we put forward automatic learning matching module (ALMM). The utilization of ALMM can realize dynamic optimization matching of MS and PAN images with different resolutions. Second, we build feature fusion residual blocks suitable for RIFRCN to optimize network parameters. Finally, we input MS image and PAN image with different resolution and complete the fusion task through ALMM, fusion, and reconstruction module. Experiments on WorldView and QuickBird datasets manifest that the proposed RIFRCNN can effectively fuse MS and PAN images, and the fusion images generated in the experiments of true and false color images show better results than the state-of-the-art technology.

Artificial Intelligence-Based Deep Fusion Model for Pan-Sharpening of Remote Sensing Images.

During the past two decades, many remote sensing image fusion techniques have been designed to improve the spatial resolution of the low-spatial-resolution multispectral bands. The main objective is fuse the low-resolution multispectral (MS) image and the high-spatial-resolution panchromatic (PAN) image to obtain a fused image having high spatial and spectral information. Recently, many artificial intelligence-based deep learning models have been designed to fuse the remote sensing images. But these models do not consider the inherent image distribution difference between MS and PAN images. Therefore, the obtained fused images may suffer from gradient and color distortion problems. To overcome these problems, in this paper, an efficient artificial intelligence-based deep transfer learning model is proposed. Inception-ResNet-v2 model is improved by using a color-aware perceptual loss (CPL). The obtained fused images are further improved by using gradient channel prior as a postprocessing step. Gradient channel prior is used to preserve the color and gradient information. Extensive experiments are carried out by considering the benchmark datasets. Performance analysis shows that the proposed model can efficiently preserve color and gradient information in the fused remote sensing images than the existing models.

Attention_FPNet: Two-Branch Remote Sensing Image Pansharpening Network Based on Attention Feature Fusion

Inspired by the impressive achievements of convolutional neural networks (CNNs) in various computer vision tasks and the effective role of attention mechanisms, this paper proposes a two-branch fusion network based on attention feature fusion (AFF) called Attention FPNet to solve the pansharpening problem. We reconstruct the spatial information of an image in the high-pass filter domain and fully consider the spatial information in the multispectral (MS) and panchromatic (PAN) images. At the same time, the input PAN image and the upsampled MS image are directly transmitted to the reconstructed image through a long skip connection. The spectral information of the PAN and MS images is considered to improve the spectral resolution of the fused image. It also supplements the loss of spatial information that may be caused by network deepening. Moreover, an AFF method is used to replace the existing simple channel concatenation method commonly used in pansharpening, which fully considers the relationship between different feature maps and improves the fusion quality. Through experiments on image datasets acquired by the Pleiades, SPOT-6 and Gaofen-2 satellites, the results show that this method can effectively fuse PAN and MS images and generate a fused image and outperforms existing methods.

Attention-Based Tri-UNet for Remote Sensing Image Pan-Sharpening

Pan-sharpening of remote sensing images is a signifi-cant method for integrating remote sensing information in the fieldof computer vision, where complementary and redundant informa-tion between multispectral (MS) images and panchromatic (PAN)images is used to generate high-resolution MS (HRMS) images.Inspired by the remarkable achievements of convolutional neuralnetworks in a variety of computer-vision tasks, we incorporatedomain-specific knowledge to design our attention-based trian-gle UNet (Tri-UNet) architecture to generate high-quality HRMSimages. The attention-based Tri-UNet is mainly divided into thefollowing three modules: 1) feature extraction; 2) feature fusion;and 3) image reconstruction. In the feature extraction step, the fea-ture extraction module simultaneously extracts spectral and spatialinformation from the MS and PAN images. The feature maps arethen fused in the feature fusion module, which makes the finalfeatureimagecontainrichspectralandspatialinformation.Finally,the image reconstruction module generates a high-resolution MSimage that uses the fused image as input. The attention mechanismis introduced into the image reconstruction module to make thenetwork focus more on key information in the feature image. Theexperimental results demonstrate that the proposed method cangenerate high-quality HRMS images. A quantitative comparisonand qualitative analysis of the experimental results indicate thatour method is superior to the existing methods.

Pan-Sharpening Based on Panchromatic Colorization Using WorldView-2

In order to overcome the lack of the multispectral image (MS) and adequately preserve the spatial information of panchromatic (PAN) image and the spectral information of MS image, this study proposes a method which adds the spectral information of the prior MS to the prior PAN during training, and only the posterior PAN is needed for predicting. Firstly, we introduce the autoencoder model based on image colorization and discuss its feasibility in the field of multi-band remote sensing image pan-sharpening. Then, the image quality evaluation functions including spatial and spectral indexes are formed as the loss function to control the image colorization model. Because the loss function contains spatial and spectral evaluation indexes, it could directly calculate the loss between the network output and the label considering characteristics of remote sensing images. Besides, the training data in our model is original PAN, this means that it is not necessary to make the simulated degraded MS and PAN data for training which is a big difference from most existing deep learning pan-sharpening methods. The new loss function including the spectral and spatial quality instead of the general MSE (mean square error), only the original PAN instead of the simulated degraded MS + PAN to be inputted, only the spectral feature instead of the direct fusion result to be learned, these three aspects change the current learning framework and optimization rule of deep learning pan-sharpening. Finally, thousands of remote sensing images from different scenes are adopted to make the training dataset to verify the effectiveness of the proposed method. In addition, we selected seven representative pan-sharpening algorithms and four widely recognized objective fusion metrics to evaluate and compare the performance on the WorldView-2 experimental data. The results show that the proposed method achieves optimal performance in terms of both the subjective visual effect and the object assessment.

PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening

This article addresses the problem of remote sensing image pan-sharpening from the perspective of generative adversarial learning. We propose a novel deep neural network-based method named pansharpening GAN (PSGAN). To the best of our knowledge, this is one of the first attempts at producing high-quality pan-sharpened images with generative adversarial networks (GANs). The PSGAN consists of two components: a generative network (i.e., generator) and a discriminative network (i.e., discriminator). The generator is designed to accept panchromatic (PAN) and multispectral (MS) images as inputs and maps them to the desired high-resolution (HR) MS images, and the discriminator implements the adversarial training strategy for generating higher fidelity pan-sharpened images. In this article, we evaluate several architectures and designs, namely, two-stream input, stacking input, batch normalization layer, and attention mechanism to find the optimal solution for pan-sharpening. Extensive experiments on QuickBird, GaoFen-2, and WorldView-2 satellite images demonstrate that the proposed PSGANs not only are effective in generating high-quality HR MS images and superior to state-of-the-art methods but also generalize well to full-scale images.

Band-Independent Encoder–Decoder Network for Pan-Sharpening of Remote Sensing Images

Pan-sharpening is a fundamental task for remote sensing image processing. It aims at creating a high-resolution multispectral (HRMS) image from a multispectral (MS) image and a panchromatic (PAN) image. In this article, a new band-independent encoder-decoder network is proposed for pan-sharpening. The network takes a single band of the MS (BMS) image, the PAN image, and the low-resolution PAN (LRPAN) image as inputs. The output of the network is the corresponding band of high-resolution MS (HRBMS) image. In this way, the network can process MS images with any number of bands. The overall structure of the network consists of two encoder-decoder modules at low-resolution and high-resolution, respectively. An auxiliary LRPAN image is used to speed up the training and improve the performance. The partly shared network and hierarchical structure for low-resolution and high-resolution enable a better fusion of features extracted from different scales. With a fast fine-tuning strategy, the trained model can be applied to images from different sensors. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pan-sharpening methods in both visual appearance and objective indexes, and the single-band evaluation results further verify the superiority of the proposed method.

Multiband Remote Sensing Image Pansharpening Based on Dual-Injection Model

Pansharpening exploits the high-frequency component (HFC) of panchromatic (PAN) images to restore the spatial-resolution of the corresponding multispectral (MS) image. In this article, a dual-injection model-based multiband remote sensing image pansharpening method is presented that focuses on how to correctly use the HFC to improve the MS image for obtaining a high-spatial resolution and MS image. The model is based on a two-step HFC injection algorithm with two different injection gains. In the first step, an HFC is reconstructed with sparse theory, and an injection gain based on the relationship between PAN and MS images is developed. Employing the previous injection gain and the reconstructed HFC on an upsampling MS image, an improved LRMS (ILRMS) image is then produced. In the second step, another injection gain based on the differences and similarities between the PAN and MS images is designed. With the help of this injection gain, the fusion image is achieved via the adaptive integration of the ILRMS image and the HFC from the PAN image. Experiments confirm that the proposed method is more effective than some popular widely used pansharpening methods.

A NSST Pansharpening method based on directional neighborhood correlation and tree structure matching

In this paper, we propose a multispectral (MS) remote sensing image pansharpening method based on non-subsampled shearlet transform (NSST). By analyzing the NSST high-frequency coefficients correlation of several datasets which are fromWorldView-2 (WV2) and Quick-Bird (QB), we verified that the high-frequency coefficients based on NSST have strong directional neighborhood correlation within the same sub-band and parent-children correlation between sub-bands in the same direction. In order to combine these two kinds of correlations, we design a type of weighted directional neighborhood templates which can be used for any number of direction sub-bands to depict the direction correlation, and use the tree structure to model the correlation between parent-children coefficients. Experiments show that the proposed method in this paper can provide a fused MS image with high spatial resolution, which can provide convenience for subsequent applications such as classification and target recognition.