Low Spatial Resolution Multispectral Research Articles

Triple disentangled network with dual attention for remote sensing image fusion

Remote sensing applications, such as detection and recognition of objects, need a large number of high spatial resolution (HR) images. As a feasible technique, image fusion is considered to produce HR images. However, most of remote sensing image fusion methods based on deep neural network (DNN) are mainly limited by the extraction of spatial and spectral features. These methods ignore the redundancy among these features, which causes spatial and spectral distortions in the fused image. In this paper, we propose a novel image fusion method based on a triple disentangled network (TDNet) with dual attention to reduce the redundancy in the extracted features. In the proposed method, it is assumed that the information in panchromatic (PAN) and low spatial resolution multispectral (LRMS) images can be encoded as the spatial, spectral, and common features. Specifically, we construct a triple-stream network to extract these features. To efficiently model the spatial and spectral information in PAN and LRMS images, local–global attention and interdependency attention are designed and integrated into the network. Then, the redundancy among these features is reduced by disentangled learning, in which these features are recombined to reconstruct the PAN and LRMS images. Besides, these features should complement each other. So, we utilize the maximal coding rate reduction to balance the redundancy and complementarity among them. Finally, all features are recombined to synthesize the high spatial resolution multispectral image. The experimental results demonstrate that the proposed TDNet has a superior performance in terms of qualitative and quantitative evaluations. The code link is https://github.com/RSMagneto/TDNet.

A Triple-Double Convolutional Neural Network for Panchromatic Sharpening.

Pansharpening refers to the fusion of a panchromatic (PAN) image with a high spatial resolution and a multispectral (MS) image with a low spatial resolution, aiming to obtain a high spatial resolution MS (HRMS) image. In this article, we propose a novel deep neural network architecture with level-domain-based loss function for pansharpening by taking into account the following double-type structures, i.e., double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the PAN image can be fully exploited and utilized to progressively inject into the low spatial resolution MS (LRMS) image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach. The code is available at https://github.com/liangjiandeng/TDNet.

Modified Dynamic Routing Convolutional Neural Network for Pan-Sharpening

Based on deep learning, various pan-sharpening models have achieved excellent results. However, most of them adopt simple addition or concatenation operations to merge the information of low spatial resolution multi-spectral (LRMS) images and panchromatic (PAN) images, which may cause a loss of detailed information. To tackle this issue, inspired by capsule networks, we propose a plug-and-play layer named modified dynamic routing layer (MDRL), which modifies the information transmission mode of capsules to effectively fuse LRMS images and PAN images. Concretely, the lower-level capsules are generated by applying transform operation to the features of LRMS images and PAN images, which preserve the spatial location information. Then, the dynamic routing algorithm is modified to adaptively select the lower-level capsules to generate the higher-level capsule features to represent the fusion of LRMS images and PAN images, which can effectively avoid the loss of detailed information. In addition, the previous addition and concatenation operations are illustrated as special cases of our MDRL. Based on MIPSM with addition operations and DRPNN with concatenation operations, two modified dynamic routing models named MDR–MIPSM and MDR–DRPNN are further proposed for pan-sharpening. Extensive experimental results demonstrate that the proposed method can achieve remarkable spectral and spatial quality.

PanDiff: A Novel Pansharpening Method Based on Denoising Diffusion Probabilistic Model

Pansharpening is a crucial image processing technique for numerous remote sensing downstream tasks, aiming to recover high spatial resolution multispectral (HRMS) images by fusing high spatial resolution panchromatic (PAN) images and low spatial resolution multispectral (LRMS) images. Most current mainstream pansharpening fusion frameworks directly learn the mapping relationships from PAN and LRMS images to HRMS images by extracting key features. However, we propose a novel pansharpening method based on the denoising diffusion probabilistic model (DDPM) called PanDiff, which learns the data distribution of the difference maps (DM) between HRMS and interpolated MS (IMS) images from a new perspective. Specifically, PanDiff decomposes the complex fusion process of PAN and LRMS images into a multi-step Markov process, and the U-Net is employed to reconstruct each step of the process from random Gaussian noise. Notably, the PAN and LRMS images serve as the injected conditions to guide the U-Net in PanDiff, rather than being the fusion objects as in other pansharpening methods. Furthermore, we propose a modal intercalibration module (MIM) to enhance the guidance effect of the PAN and LRMS images. The experiments are conducted on a freely available benchmark dataset, including GaoFen-2, QuickBird, and WorldView-3 images. The experimental results from the fusion and generalization tests effectively demonstrate the outstanding fusion performance and high robustness of PanDiff. Fig. 1 depicts the results of the proposed method performed on various scenes. Additionally, the ablation experiments confirm the rationale behind PanDiff’s construction.

A Novel Spatial Fidelity With Learnable Nonlinear Mapping for Panchromatic Sharpening

The purpose of panchromatic sharpening, i.e., pansharpening, is to fuse a low spatial resolution multispectral (LRMS) image with a high spatial resolution panchromatic (PAN) image, aiming to obtain a high spatial resolution multispectral (HRMS) image. Pansharpening models based on variational optimization consist of a spectral fidelity term, a spatial fidelity term, and a regularization term. Most of the methods assume that the existing PAN image and the homologous HRMS image satisfy the global or local linear relationship, which could be far from the real case, thus causing sub-optimal performance. Inspired by the nonlinear mapping ability of machine learning (ML) techniques, we propose a novel spatial fidelity term with learnable nonlinear mapping (LNM-SF), which trains an implicit functional operator via a specifically designed convolutional neural network (CNN) and efficiently constructs the nonlinear relationship between the known PAN and the latent HRMS images. Relying upon the above description of the spatial fidelity term, a new variational model with a learnable nonlinear mapping in the spatial fidelity term for pansharpening, named LNM-PS, is simply integrated by the conventional spectral fidelity term into the proposed LNM-SF. To effectively solve the resulting optimization problem, we develop an alternating direction method of multipliers (ADMM)-based algorithm with the fast iterative shrinkage-thresholding algorithm (FISTA) as inner solver. Extensive numerical experiments on different datasets, assessing the performance both at reduced-resolution and full-resolution, show the superiority of the proposed LNM-PS method. The code is available at https://github.com/liangjiandeng/-LNM-PS.

Multiscale Dual-Domain Guidance Network for Pan-Sharpening

The goal of pan-sharpening is to produce a high-spatial-resolution multi-spectral (HRMS) image from a low-spatial-resolution multi-spectral (LRMS) counterpart by super-resolving the LRMS one under the guidance of a texture-rich panchromatic (PAN) image. Existing research has concentrated on using spatial information to generate HRMS images, but has neglected to investigate the frequency domain, which severely restricts the performance improvement. In this work, we propose a novel pan-sharpening approach, named Multi-Scale Dual-Domain Guidance Network (MSDDN) by fully exploring and exploiting the distinguished information in both the spatial and frequency domains. Specifically, the network is inborn with multi-scale U-shape manner and composed by two core parts: a spatial guidance sub-network for fusing local spatial information and a frequency guidance sub-network for fusing global frequency domain information and encouraging dual-domain complementary learning. In this way, the model can capture multi-scale dual-domain information to help it generate high-quality pan-sharpening results. Employing the proposed model on different datasets, the quantitative and qualitative results demonstrate that our method performs appreciatively against other state-of-the-art approaches and comprises a strong generalization ability for real-world scenes. The source code is available at https://github.com/alexhe101/MSDDN.

DiTBN: Detail Injection-Based Two-Branch Network for Pansharpening of Remote Sensing Images

Pansharpening is one of the main research topics in the field of remote sensing image processing. In pansharpening, the spectral information from a low spatial resolution multispectral (LRMS) image and the spatial information from a high spatial resolution panchromatic (PAN) image are integrated to obtain a high spatial resolution multispectral (HRMS) image. As a prerequisite for the application of LRMS and PAN images, pansharpening has received extensive attention from researchers, and many pansharpening methods based on convolutional neural networks (CNN) have been proposed. However, most CNN-based methods regard pansharpening as a super-resolution reconstruction problem, which may not make full use of the feature information in two types of source images. Inspired by the PanNet model, this paper proposes a detail injection-based two-branch network (DiTBN) for pansharpening. In order to obtain the most abundant spatial detail features, a two-branch network is designed to extract features from the high-frequency component of the PAN image and the multispectral image. Moreover, the feature information provided by source images is reused in the network to further improve information utilization. In order to avoid the training difficulty for a real dataset, a new loss function is introduced to enhance the spectral and spatial consistency between the fused HRMS image and the input images. Experiments on different datasets show that the proposed method achieves excellent performance in both qualitative and quantitative evaluations as compared with several advanced pansharpening methods.

Pansharpening via Triplet Attention Network With Information Interaction

Pansharpening aims to obtain high spatial resolution multispectral (MS) images by fusing the spatial and spectral information in low spatial resolution (LR) MS and panchromatic (PAN) images. Recently, deep neural network (DNN) based pansharpening methods have been advanced extensively. Although most DNN-based methods show good performance, it is difficult for them to preserve the spatial details in the fused image. In this article, we propose a new pansharpening method based on a triplet attention network with information interaction to efficiently enhance the spatial and spectral information in the fused image. First, different attention mechanisms are designed to model the spatial and spectral feature properties in LR MS and PAN images. Then, the complementarity among different feature maps is enhanced by information interaction, which promotes the compatibility of features from subnetworks. Finally, we utilize a graph attention module to capture the similarity within feature maps. According to the graph, the informative feature maps are selected to provide more details for the reconstruction of the fused image. Extensive experiments on QuickBird and GeoEye-1 satellite datasets show that the proposed method can produce competitive fused images when compared with some state-of-the-art methods.

Pan-Sharpening Based on Transformer With Redundancy Reduction

Pan-sharpening methods based on deep neural network (DNN) have produced the state-of-the-art results. However, the common information in the panchromatic (PAN) image and the low spatial resolution multispectral (LRMS) image is not sufficiently explored. As PAN and LRMS images are collected from the same scene, there exists some common information among them, in addition to their respective unique information. The direct concatenation of extracted features leads to some redundancy in the feature space. To reduce the redundancy among features and exploit the global information in source images, we proposed a novel pan-sharpening method by combining the convolution neural network and transformer. Specifically, PAN and LRMS images are encoded as unique features and common features by the subnetworks consisting of convolution blocks and transformer blocks. Then, the common features are averaged and combined with unique features from source images for the reconstruction of the fused image. To extract accurate common features, the equality constraint is imposed on them. Experimental results show that the proposed method outperforms the state-of-the-art methods on both reduced-scale and full-scale datasets. The source code is available at <uri>https://github.com/RSMagneto/TRRNet</uri>.

Fast and High-Quality Blind Multi-Spectral Image Pansharpening

Blind pansharpening addresses the problem of generating a high spatial-resolution multi-spectral (HRMS) image given a low spatial-resolution multi-spectral (LRMS) image with the guidance of its associated spatially misaligned high spatial-resolution panchromatic (PAN) image without parametric side information. In this paper, we propose a fast approach to blind pansharpening and achieve state-of-the-art image reconstruction quality. Typical blind pansharpening algorithms are often computationally intensive since the blur kernel and the target HRMS image are often computed using iterative solvers and in an alternating fashion. To achieve fast blind pansharpening, we decouple the solution of the blur kernel and of the HRMS image. First, we estimate the blur kernel by computing the kernel coefficients with minimum total generalized variation that blur a downsampled version of the PAN image to approximate a linear combination of the LRMS image channels. Then, we estimate each channel of the HRMS image using local Laplacian prior to regularize the relationship between each HRMS channel and the PAN image. Solving the HRMS image is accelerated by both parallelizing across the channels and by fast numerical algorithms for each channel. Due to the fast scheme and the powerful priors we used on the blur kernel coefficients (total generalized variation) and on the cross-channel relationship (local Laplacian prior), numerical experiments demonstrate that our algorithm outperforms state-of-the-art model-based counterparts in terms of both computational time and reconstruction quality of the HRMS images.

Pan-Sharpening via Deep Locally Linear Embedding Residual Network

The goal of pan-sharpening tasks is to fuse panchromatic (PAN) images and low-spatial-resolution (LR) multispectral (MS) images for the purpose of aggregating texture and spectral information. Although traditional embedding-based pan-sharpening methods achieve competitive results, they are limited by the shallow network and not suitable for large-scale datasets. In this study, we design a novel multiscale locally linear embedding residual network (LLERN) that consists of two phases: the spectral preservation phase and the structural preservation phase. As the pretreatment of the structural preservation network, the spectral preservation network aims to upscale the LR MS image while retaining spectral information. The proposed locally linear embedding residual block (LLERB) in the structural preservation phase can search for similar sparse patches from the PAN image space and embed the corresponding local geometric relationship into the residual space to enhance the MS image. Extensive experiments suggest that the proposed LLERN outperforms state-of-the-art methods from visual and quantitative perspectives, and confirm the assumption that LR image patches and residual image patches in a local region share a similar manifold structure, which can be used to guide deep-learning modeling with improved interpretability. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jiaming-wang/LLERN</uri> .

A differential information residual convolutional neural network for pansharpening

Deep learning based methods are the state-of-the-art in panchromatic (PAN)/multispectral (MS) fusion (which is generally called “pansharpening”). In this paper, to solve the problem of the insufficient spatial enhancement in most of the existing deep learning based pansharpening methods, we propose a novel pansharpening method based on a residual convolutional neural network (RCNN). Differing from the existing deep learning based pansharpening methods that are mainly devoted to designing an effective network, we make novel changes to the input and the output of the network and propose a simple but effective mapping strategy. This strategy involves utilizing the network to map the differential information between the high spatial resolution panchromatic (HR-PAN) image and the low spatial resolution multispectral (LR-MS) image to the differential information between the HR-PAN image and the high spatial resolution multispectral (HR-MS) image, which is called the “differential information mapping strategy”. Moreover, to further boost the spatial information in the fusion results, the proposed method makes full use of the LR-MS image and utilizes the gradient information of the up-sampled LR-MS image (Up-LR-MS) as auxiliary data to assist the network. Furthermore, an attention module and residual blocks are incorporated in the proposed network structure to maximize the ability of the network to extract features. Experiments on four data sets collected by different satellites confirm the superior performance of the proposed method compared to the state-of-the-art pansharpening methods.

Superpixel guided structure sparsity for multispectral and hyperspectral image fusion over couple dictionary

This paper proposed a hyperspectral (HS) and multispectral (MS) image fusion method based on superpixel guided structure sparsity and couple dictionary (SGSSCD). It is assumed that the pixels in a homogeneous area of MS image are similar due to the consistent spatial consistency. Superpixel technique is used to find the similar pixels in MS image by considering the region homogeneity. Then these pixels naturally share the same atoms in low spectral resolution dictionary. In order to capture the similarity prior, structural sparsity is employed to find more efficient coding of the similar pixels in MS image over low spectral resolution dictionary. Finally, high spatial resolution HS image can be produced by combining the codes of MS image with high spectral resolution dictionary. Besides, the couple dictionary is learned from HS and low spatial resolution MS images to ensure the spectral correspondence, which can further improve the quality of fusion results. The experimental results on different datasets demonstrate the effectiveness of the proposed method when compared with some existing methods.

Incorporating an Adaptive Image Prior Model Into Bayesian Fusion of Multispectral and Panchromatic Images

Reconstruction of a sharpened multispectral (MS) image from its coarser measurements, namely the low spatial resolution MS and panchromatic (Pan) images, is a severely illposed inverse problem which requires the definition of an appropriate prior model. This letter incorporates an adaptive Markov random field (MRF)-based prior model into a Bayesian framework to recover the desired MS image. The proposed MRF-based prior model combines the high frequency details of the Pan image with the spectral relation between the bands of the MS image into a single energy function. Consequently, unlike most pansharpening methods, not directly injecting the spatial information of the Pan image into the fused product, the proposed method offers a fused product with minimum spectral distortion, along with perfectly enhanced spatial resolution. Visual and quantitative assessments of the fused products of the proposed method compared to those of some famous pansharpening methods prove the superiority of the proposed method.

A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening

Pan-sharpening is a fundamental and significant task in the field of remote sensing imagery processing, in which high-resolution spatial details from panchromatic images are employed to enhance the spatial resolution of multispectral (MS) images. As the transformation from low spatial resolution MS image to high-resolution MS image is complex and highly nonlinear, inspired by the powerful representation for nonlinear relationships of deep neural networks, we introduce multiscale feature extraction and residual learning into the basic convolutional neural network (CNN) architecture and propose the multiscale and multidepth CNN for the pan-sharpening of remote sensing imagery. Both the quantitative assessment results and the visual assessment confirm that the proposed network yields high-resolution MS images that are superior to the images produced by the compared state-of-the-art methods.

Improved Pansharpening with Un-Mixing of Mixed MS Sub-Pixels near Boundaries between Vegetation and Non-Vegetation Objects

Pansharpening is an important technique that produces high spatial resolution multispectral (MS) images by fusing low spatial resolution MS images and high spatial resolution panchromatic (PAN) images of the same area. Although numerous successful image fusion algorithms have been proposed in the last few decades to reduce the spectral distortions in fused images, few of these take into account the spectral distortions caused by mixed MS sub-pixels (MSPs). Typically, the fused versions of MSPs remain mixed, although some of the MSPs correspond to pure PAN pixels. Due to the significant spectral differences between vegetation and non-vegetation (VNV) objects, the fused versions of MSPs near VNV boundaries cause blurred VNV boundaries and significant spectral distortions in the fused images. In order to reduce the spectral distortions, an improved version of the haze- and ratio-based fusion method is proposed to realize the spectral un-mixing of MSPs near VNV boundaries. In this method, the MSPs near VNV boundaries are identified first. The identified MSPs are then defined as either pure vegetation or non-vegetation pixels according to the categories of the corresponding PAN pixels. Experiments on WorldView-2 and IKONOS images of urban areas using the proposed method yielded fused images with significantly clearer VNV boundaries and smaller spectral distortions than several other currently-used image fusion methods.

An Edge Preserving Multiresolution Fusion: Use of Contourlet Transform and MRF Prior

In this paper, we propose a new approach for multiresolution fusion using contourlet transform (CT). The method is based on modeling the low spatial resolution (LR) and high spectral resolution multispectral (MS) image as the degraded and noisy version of their high spatial resolution version. Since this is an ill-posed problem, it requires regularization in order to obtain the final solution. In this paper, we first obtain the initial estimate of the fused image from the available MS image and the panchromatic (Pan) image by using the CT domain learning. Since CT provides better directional edges, the initial estimate has better edge details. Using the initial estimate, we obtain the degradation that accounts for the aliasing between the LR MS image and fused image. Regularization is carried out by modeling the texture of the final fused image as a homogeneous Markov random field (MRF) prior, where the MRF parameter is estimated using the initial estimate. The use of MRF prior on the final fused image takes care of the spatial dependencies among the pixels. A simple gradient-based optimization technique is used to obtain the final fused image. Although we use homogeneous MRF, the proposed approach preserves the edges in the final fused image by retaining the edges from the initial estimate and by carrying out the optimization on nonedge pixels only. Therefore, the advantage of the proposed method lies in preserving the discontinuities without using the discontinuity preserving prior, thus avoiding the use of computationally taxing optimization techniques for regularization purposes. In addition, the proposed method causes minimum spectral distortion since it learns the texture using contourlet coefficients and does not use actual Pan image pixel intensities. We demonstrate the effectiveness of our approach by conducting the experiments using subsampled and nonsubsampled CT on different data sets captured using Ikonos-2, Quickbird, and Worldview-2 satellites.

Multiresolution Image Fusion: Use of Compressive Sensing and Graph Cuts

In this paper, we propose a new approach for multiresolution fusion, i.e., obtaining a high spatial and spectral resolution multispectral (MS) image using the available low spatial resolution MS and the high spatial resolution Panchromatic (Pan) image. Our approach is based on the idea of compressive sensing (CS) and graph cuts. Assuming that both the MS and Pan images have the same sparseness, a close approximation to the MS image is obtained from the Pan image using the theory of compressive sensing and l1 minimization. We then use regularization framework to obtain fused image. The low resolution (LR) MS image is modeled as degraded and noisy version of fused image in which degradation matrix entires estimated using the close approximation are used. The regularization is carried out by using truncated quadratic smoothness prior which takes care of preservation of the discontinuities in the fused image. A suitable energy function is then formed consisting of data fitting term and prior term. Minimization of the energy function is carried out using a computationally efficient graph cuts optimization to obtain final fused image. Advantage of our approach is that the Pan and MS images need not be registered. This is because, we are not directly using the Pan digital numbers to derive the fused image. The effectiveness of the proposed method is illustrated by conducting experiments on real satellite images. Subjective and quantitative comparison of the proposed method with the state-of-the-art approaches indicates efficacy of our approach.

Fusion of multispectral and panchromatic images via sparse representation and local autoregressive model

In this paper, a novel model-based pan-sharpening method via sparse representation and local autoregressive (AR) model is proposed. To recover the high-resolution multispectral (HRMS) image from the observed images, we impose sparsity prior on the unknown HRMS image in the restoration model. The quality of the recovered HRMS image depends on the employed sparse domain. Hence, a new sparse representation model for the HRMS image is constructed, in which we suppose that the low-frequency and high-frequency components of the HRMS image can be sparsely represented by a spectral dictionary and a spatial-detail dictionary respectively. The spectral dictionary and spatial-detail dictionary are learned from the source images: low-spatial-resolution multispectral (LRMS) image and high-spatial-resolution panchromatic (HRP) image adaptively. Additionally, local autoregressive (AR) model is employed to improve the spatial structure of the HRMS image patch. Firstly, a set of AR model parameters are learned from the PAN image patches. Then, the local spatial structure of a given HRMS image patch is regularized by an AR model with the learned parameters. By solving the l1 -norm optimization problem, the HRMS image can be well reconstructed. Experiments are carried out on very high-resolution QuickBird and GeoEye-1 images. In the simulated and real experiments, our proposed method demonstrates its good performance in terms of visual analysis and quantitative evaluation.