High-resolution Multispectral Research Articles

A novel cross fusion model with fine-grained detail reconstruction for remote sensing image pan-sharpening

ABSTRACT Pan-sharpening aims to obtain high resolution multispectral (HRMS) images by integrating the information in the panchromatic and multispectral images. Existing pan-sharpening methods have demonstrated impressive sharpening performance. However, these methods inherently overlook the complementary characteristics and interaction between diverse source images, resulting in sharpened outcomes accompanied by distortion. To solve the above problems, we construct a novel cross fusion model with fine-grained detail reconstruction from the perspective of frequency-domain. The motivation of the model is twofold: (1) to reconstruct spatial detail representations from diverse source images, laying the foundation for the generation of fine details in the subsequent fused images; and (2) to enhance the interaction between diverse source features during the fusion process in order to attain high-fidelity fusion outcomes. Based on the theoretical model, we develop a frequency-spectral dual domain cross fusion network (CF2N) utilizing the deep learning technique. Consequently, the CF2N consist of two main stages, namely frequency-domain dominated detail reconstruction (FD2R) and frequency-spectral cross fusion (FSCF). Specifically, a more reasonable reconstruction of fine frequency details in HRMS can be achieved by performing adaptive weighted fusion of frequency details in the FD2R stage. Furthermore, the FSCF module, which seamlessly integrates frequency- and spectral-domain details in a highly interactive cross fusion manner. As a result, the CF2N possesses the capability to attain high frequency-spectral fidelity results with excellent interpretability. Extensive experiments show the superior performance of ours over state of the art, while maintaining high efficiency. All implementations of this work will be published at our website.

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Multispectral and hyperspectral image fusion (MS/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods.

Rethinking Pan-Sharpening in Closed-Loop Regularization.

It is generally known that pan-sharpening is fundamentally a PAN-guided multispectral (MS) image super-resolution problem that involves learning the nonlinear mapping from low-resolution (LR) to high-resolution (HR) MS images. Since an infinite number of HR-MS images can be downsampled to produce the same corresponding LR-MS image, learning the mapping from LR-MS to HR-MS image is typically ill-posed and the space of the possible pan-sharpening functions can be extremely large, making it difficult to estimate the optimal mapping solution. To address the above issue, we propose a closed-loop scheme that learns the two opposite mapping including the pan-sharpening and its corresponding degradation process simultaneously to regularize the solution space in a single pipeline. More specifically, an invertible neural network (INN) is introduced to perform a bidirectional closed-loop: the forward operation for LR-MS pan-sharpening and the backward operation for learning the corresponding HR-MS image degradation process. In addition, given the vital importance of high-frequency textures for the Pan-sharpened MS images, we further strengthen the INN by designing a specified multiscale high-frequency texture extraction module. Extensive experimental results demonstrate that the proposed algorithm performs favorably against state-of-the-art methods qualitatively and quantitatively with fewer parameters. Ablation studies also verify the effectiveness of the closed-loop mechanism in pan-sharpening. The source code is made publicly available at https://github.com/manman1995/pan-sharpening-Team-zhouman/.

A novel pansharpening model based on two parallel network architectures

ABSTRACT Pansharpening is an important technology for obtaining high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images. Although many pansharpening models have emerged by taking advantage of deep learning (DL) technology, there remains a pressing need to further assess pansharpening accuracy and stability when LRMS images with complex land-cover types. What’s more, these models often overlook the exploitation of PAN images’ inherent high-frequency information. To address these issues, we propose a pansharpening model combining multi-level and multi-scale network architectures. The multi-level network architecture is used to build spatial-spectral dependence on LRMS-PAN pairs, and strengthen the network’s feature capture capability by keeping the multi-level texture details. The multi-scale architecture is subsequently used to extract the spatial structure and deep texture of the PAN images at different scales. Downsampled experiments and real experiments in four standard datasets show that the proposed model achieves a state-of-the-art performance.

Pansharpening Based on Multimodal Texture Correction and Adaptive Edge Detail Fusion

Pansharpening refers to the process of fusing multispectral (MS) images with panchromatic (PAN) images to obtain high-resolution multispectral (HRMS) images. However, due to the low correlation and similarity between MS and PAN images, as well as inaccuracies in spatial information injection, HRMS images often suffer from significant spectral and spatial distortions. To address these issues, a pansharpening method based on multimodal texture correction and adaptive edge detail fusion is proposed in this paper. To obtain a texture-corrected (TC) image that is highly correlated and similar to the MS image, the target-adaptive CNN-based pansharpening (A-PNN) method is introduced. By constructing a multimodal texture correction model, intensity, gradient, and A-PNN-based deep plug-and-play correction constraints are established between the TC and source images. Additionally, an adaptive degradation filter algorithm is proposed to ensure the accuracy of these constraints. Since the TC image obtained can effectively replace the PAN image and considering that the MS image contains valuable spatial information, an adaptive edge detail fusion algorithm is also proposed. This algorithm adaptively extracts detailed information from the TC and MS images to apply edge protection. Given the limited spatial information in the MS image, its spatial information is proportionally enhanced before the adaptive fusion. The fused spatial information is then injected into the upsampled multispectral (UPMS) image to produce the final HRMS image. Extensive experimental results demonstrated that compared with other methods, the proposed algorithm achieved superior results in terms of both subjective visual effects and objective evaluation metrics.

Bayesian decision based fusion algorithm for remote sensing images

Remote sensing image fusion is dedicated to obtain a high-resolution multispectral (HRMS) image without spatial or spectral distortion compared to the single source image. In this paper, a novel fusion algorithm based on Bayesian estimation for remote sensing images is proposed from the new perspective of risk decisions. In this study, an observation model based on Bayesian estimation for remote sensing image fusion is constructed. Three categories of probabilities including prior, conditional and posterior probabilities are calculated after an intensity-hue-saturation (IHS) transformation is applied to the original low-resolution MS image. To obtain the desired HRMS image, with the corrected posterior probability, a fusion rule based on Bayesian decisions is designed to estimate which pixels to select from the panchromatic (PAN) image and the intensity component of the MS image. The selected pixels constitute a new component that will participate in an IHS inverse transformation to yield the fused image. Extensive experiments were performed on the Pleiades, WorldView-3, and IKONOS datasets, and the results demonstrate the effectiveness of the proposed method.

Bayesian fusion of multispectral and panchromatic images using a multi-mode and multiorder gradient tensor-based l 1 / 2 sparse model

ABSTRACT In this letter, based on the tensor representation modelling, we propose a multi-mode and multi-order gradient tensor-based non-convex model (M2GTNM) for Bayesian fusion of multispectral (MS) and panchromatic (Pan) images, which aims at producing the high-resolution MS (HRMS) images. Specifically, by modelling the MS image as the order-3 tensor, we mainly develop the multi-mode and multi-order gradient tensor sparse priors of MS image for fusion. For the spectral preservation of low-resolution MS (LRMS) image, the spectral fidelity constraint between HRMS and LRMS images is imposed. For the spatial-mode prior modelling, the multi-order spatial gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and Pan images is particularly imposed. Moreover, for the spectral-mode prior modelling, the spectral gradient tensor-based non-convex l 1 / 2 sparse prior between HRMS and upsampled LRMS images is further imposed. Then, we apply the alternating direction method of multipliers to optimize the proposed model. Finally, the reduced-scale and full-scale fusion experiments both validate the effectiveness of M2GTNM method.

DUCD: Deep Unfolding Convolutional-Dictionary network for pansharpening remote sensing image

The goal of pansharpening methods is to complement the spectral and spatial information contained in Multi-spectral (MS) and panchromatic (PAN) images to obtain the desired High-resolution multispectral (HRMS) image. The existing majority of pansharpening methods either extract feature information separately from the MS image and PAN image, or extract feature information after concatenating the MS image and PAN image. However, the entire extraction process lacks the utilization of complementary information and tends to generate redundant information, thereby leading to the loss of certain important information during the extraction process, which in turn affects the overall performance. In order to better utilize the complementary information between the MS image and PAN image and enhance the interpretability of the network, we propose the Deep Unfolding Convolutional-Dictionary Network (DUCD) for pansharpening in this paper. This network fully integrates complementary information between the MS image and PAN image to generate the final fused image. The entire network structure consists of two parts: The encoder and the decoder. In the encoder part of the network, we clarify the common and unique feature information between MS and PAN images by constructing an observation model. Simultaneously, we use the approximate gradient algorithm to continuously optimize the model and iteratively unfold it into a deep network structure. In the decoder part of the network, we concatenate the obtained common and specific information from MS and PAN images and pass them through convolutional and activation layers. Subsequently, they are input into the introduced Frequency Domain-based Transformer (FDT) module and an information-lossless inversible neural network(INN). This provides a more efficient method for establishing long-range dependency relationships between feature extraction and feature fusion. To demonstrate the effectiveness of our proposed method, we conduct extensive experiments on three benchmark datasets QB, GF2 and WV3. Experimental results show that our method outperforms the current SOTA Pansharpening methods in terms of performance.

Transformer-based dual path cross fusion for pansharpening remote sensing images

ABSTRACT The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods.

Pan-sharpening via intrinsic decomposition knowledge distillation

Existing deep-learning-based pan-sharpening strategies mainly involve the fusion of panchromatic and multispectral (MS) information at both the pixel and feature levels. In this paper, we hypothesize that the MS image can be expressed as the multiplication of reflectance and illumination components, and that the reflection components of low-resolution (LR) MS and high-resolution (HR) MS images are invariant. Here, the spectral reflection component can effectively describe the spectral response of an object, while the illumination component can effectively describe its texture. Based on this hypothesis, we propose a novel and concise pan-sharpening framework called intrinsic decomposition knowledge distillation. Specifically, the teacher network decomposes the HR MS image into reflectance and illumination components, which are then combined in the student network with the reflectance component and the enhanced illumination component from LR MS to reconstruct the pan-sharpened image. To approximate the component distributions from the teacher network, we introduce a novel three-stage knowledge distillation strategy that can transfer knowledge about the relationships between components and constrain the student network. Our quantitative and qualitative comparisons demonstrate the reasonableness of our hypothesis and the effectiveness of our proposed method in significantly improving perception quality.

CrossDiff: Exploring Self-SupervisedRepresentation of Pansharpening via Cross-Predictive Diffusion Model.

Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS images. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation for pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional Denoising Diffusion Probabilistic Model (DDPM). While in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS images, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite datasets. Code is available at https://github.com/codgodtao/CrossDiff.

A variational driven optimization framework for pansharpening of multispectral images

ABSTRACT Pansharpening is an established remote sensing image fusion technique that yields a high-resolution multispectral (HRMS) image. Despite the fact that the advanced technologies like sparse coding and deep learning have achieved a remarkable improvement in solving the pansharpening problem, a unified model is required to further enhance the fusion quality. The variational optimisation (VO) mechanism has gained interest of most of the researchers in recent years. In this article, pansharpening is designated as a constrained optimisation problem with a data generative term and two regularisers to ameliorate spatial details and spectral information. The gradient information is exploited to impart the spatial details from the panchromatic image to the fused image. The correlation among multispectral image bands inspired to promote the spectral quality of the HRMS image as well as to reduce the distortion. Consequently, the optimisation problem is efficiently solved for the required HRMS image using the operator splitting approach. The extensive experiments performed in accordance with the well-known protocols rationalise that the proposed model outperforms most of the state-of-the-art methods in terms of objective metrics and visual outcomes.

A Triple-Double Convolutional Neural Network for Panchromatic Sharpening.

Pansharpening refers to the fusion of a panchromatic (PAN) image with a high spatial resolution and a multispectral (MS) image with a low spatial resolution, aiming to obtain a high spatial resolution MS (HRMS) image. In this article, we propose a novel deep neural network architecture with level-domain-based loss function for pansharpening by taking into account the following double-type structures, i.e., double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the PAN image can be fully exploited and utilized to progressively inject into the low spatial resolution MS (LRMS) image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach. The code is available at https://github.com/liangjiandeng/TDNet.

Triple‐loss driven generative adversarial network for pansharpening

AbstractPansharpening aims at fusing a panchromatic (PAN) image and a low‐resolution multispectral (LRMS) image into a high‐resolution multispectral (HRMS) image. In recent years, GAN‐based pansharpening methods have achieved excellent results, but they suffer from inadequate feature preservation and unstable training. To address these issues, a novel GAN‐based model named TriLossGAN is proposed. This method constructs three loss components with the help of the generator and the dual‐discriminator, which are calculated in both the original spatial domain and the transform domain to better preserve high‐frequency and low‐frequency information in the fused image. Additionally, a new training strategy is designed to stabilize the training process. In extensive experiments, the proposed method achieved satisfactory results on three datasets with QNR values of 0.9584 on GaoFen‐2, 0.9601 on QuickBird, and 0.9138 on WorldView‐3. Qualitative and quantitative comparisons demonstrate that TriLossGAN outperforms other state‐of‐the‐art methods.

Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion for remote sensing image

ABSTRACT Remote sensing image (RSI) with concurrently high spatial, temporal, and spectral resolutions cannot be produced by a single sensor. Multisource RSI fusion is a convenient technique to realize high spatial resolution multispectral (MS) images (spatial spectral fusion, i.e. SSF) and high temporal and spatial resolution MS images (spatiotemporal fusion, i.e. STF). Currently, deep learning-based fusion models can only implement SSF or STF, lacking models that perform both SSF and STF. Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion (BAPGF) for RSI are proposed to implement both SSF and STF, namely BPF-MGAN. A bidirectional adaptive-stage feature extraction architecture in fine-scale-to-coarse-scale and coarse-scale-to-fine-scale modes is introduced. The designed BAPGF introduces a previous fusion result-oriented cross-stage-level dual-residual attention fusion strategy to enhance critical information and suppress superfluous information. Adaptive resolution U-shaped discriminators are implemented to feed multiresolution context into the generator. A generalized multitask loss function unlimited by no-reference images is developed to strengthen the model via constraints on the multiscale feature, structural, and content similarities. The BPF-MGAN model is validated on SSF datasets and STF datasets. Compared with the state-of-the-art SSF and STF models, results demonstrate the superior performance of the proposed BPF-MGAN model in both subjective and objective evaluations.

FrMLNet: Framelet-Based Multilevel Network for Pansharpening.

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

A Spectral Enhancement Method Based on Remote-Sensing Images for High-Speed Railways

This paper proposes a pansharpening model in order to obtain remote-sensing images with high spatial resolution and high spectral resolution. Based on a generic component substitution (CS) fusion framework, the model utilizes the difference between the high-frequency component of the panchromatic (PAN) image and the high-frequency component of the luminance (L) image to express the missing spatial detail information of the ideal high-resolution multispectral (HRMS) image. A rolling guidance filter (RGF) is used in this framework to achieve the effective extraction of high-frequency information from remote-sensing images while reducing the spectral distortion of subsequent operations. The modulation transfer function (MTF) values of the sensor are also applied to the selection of adaptive weighting coefficients to further improve the spectral fidelity of the fused images. At the same time, the choice of suitable interpolation and gain coefficients improves the generalizability of the model while reducing spectral and spatial distortions. Finally, the use of a guided filter (GF) also greatly improves the quality of the fused image. The experimental results show that the model can effectively improve the spatial resolution for foreign objects at the perimeter of high-speed railways, while also ensuring the color fidelity of foreign objects such as colored steel tiles.

A deep unrolling pansharpening method based on spectral consistency and double spatial priors

ABSTRACT In recent years, deep learning (DL) pansharpening methods have become the most popular solution for acquiring high-resolution multispectral (HRMS) images. However, most DL-based methods design the task as black-box network architecture, ignoring the physical meaning of individual network modules. Such a network structure is difficult to extract sufficient spatial details or lacks interpretability. To address this problem, we propose a neural network pansharpening method with interpretable deep spatial detail injection based on the assumption of spectral consistency and double spatial detail priors in HRMS images. First, we assume that the HRMS image consists of a spectral and spatial component together. Spectral and partial spatial details are obtained by upsampling the multispectral (MS) images. Next, a variational model is constructed for image fusion from the LRMS image and the panchromatic (PAN) image to the HRMS image.The alternating direction multiplier method (ADMM) algorithm is then used to optimally solve the variational model to obtain the spatial detail that needs to be injected into the MS image. Finally, the model optimization solving process is expanded into a corresponding neural network structure using the idea of deep unrolling. Each network module corresponds to the solving step of the iterative algorithm. Our proposed pansharpening method is therefore interpretable in terms of spatial detail injection. In our experiments, we find that the proposed method achieves an optimal balance between spectral and spatial quality, and has a strong generalization capability over different datasets, demonstrating the superiority of the proposed pansharpening method.

Intensity mixture and band-adaptive detail fusion for pansharpening

Pansharpening aims to sharpen a low-resolution multispectral (MS) image through a high-resolution single-channel panchromatic (PAN) image to obtain a high-resolution multi-spectral (HRMS) image. However, low correlation between the PAN and MS images, as well as the inaccurate detail injection for each band of MS image are the key problems causing spectral and spatial distortions in pansharpening. To address these issues, a new pansharpening method based on the intensity mixture and band-adaptive detail fusion is proposed. To obtain a mixed-intensity image (T) that has a high correlation with the MS image and maintain the gradient information of the PAN image, the intensity mixture model is constructed by establishing the intensity and gradient constraints between T and the source images. As it is hard to obtain a proper degradation filter in the model, a filter estimation algorithm is designed by the distribution alignment. To inject the details that match the point spread function of the sensor, a band-adaptive detail fusion algorithm is presented to fuse the details extracted from T with those from the MS image for each band. Furthermore, as there are far fewer details in the MS image than in T, a detail enhancement algorithm is proposed to enhance the details proportionally. The final HRMS image is obtained by injecting the fused details into the upsampled MS image. Extensive experiments show that the proposed method can efficiently achieve the best results in fusion quality compared to state-of-the-art methods. The code is availabe at https://github.com/yotick/IMBD.

Pan-sharpening of remote sensing images based on gradient projection and cross fusion

ABSTRACT Pan-sharpening is the fusion of panchromatic (PAN) image and multispectral (MS) image through certain rules to generate high-resolution multispectral (HRMS) image with vivid spatial details and uniform spectral distribution. It has become an important technology in remote sensing image processing. Recently, convolutional neural networks based on deep learning have achieved remarkable results in the field of pan-sharpening. Guided by this method, this paper proposes a pan-sharpening network PSAM-NET for remote sensing images based on depth expansion combined with Cross-Attention Fusion. It consists of gradient projection and the main image fusion module. The gradient projection mainly generates the fusion module by stacking low-resolution multispectral images and panchromatic images alternately, and two deep prior regularized optimization problem formulas are respectively solved by the gradient projection algorithm. The other principal image fusion module realizes double branch fusion, which is mainly composed of cross attention fusion and channel attention fusion, to produce an excellent fusion effect. The simulation and real data experiments were performed on the standard datasets WV2, GF-2, and QB. The qualitative analysis and quantitative comparison with the classical pan-sharpening method proved that the spatial information of the image obtained by this method is complete. The spectral distribution is uniform, and the evaluation index also has some advantages.