Low-resolution Multispectral Research Articles

PanCSC-Net: A Model-Driven Deep Unfolding Method for Pansharpening

Recently, deep learning (DL) approaches have been widely applied to the pansharpening problem, which is defined as fusing a low-resolution multispectral (LRMS) image with a high-resolution panchromatic (PAN) image to obtain a high-resolution multispectral (HRMS) image. However, most DL-based methods handle this task by designing black-box network architectures to model the mapping relationship from LRMS and PAN to HRMS. These network architectures always lack sufficient interpretability, which limits their further performance improvements. To address this issue, we adopt the model-driven method to design an interpretable deep network structure for pansharpening. First, we present a new pansharpening model using the convolutional sparse coding (CSC), which is quite different from the current pansharpening frameworks. Second, an alternative algorithm is developed to optimize this model. This algorithm is further unfolded to a network, where each network module corresponds to a specific operation of the iterative algorithm. Therefore, the proposed network has clear physical interpretations, and all the learnable modules can be automatically learned in an end-to-end way from the given dataset. Experimental results on some benchmark datasets show that our network performs better than other advanced methods both quantitatively and qualitatively.

PMACNet: Parallel Multiscale Attention Constraint Network for Pan-Sharpening

Pan-sharpening, a task involving information fusion, entails merging panchromatic (PAN) images with high spatial resolution and low-resolution multispectral (LRMS) images in order to obtain high-resolution multispectral (HRMS) images. Due to deep learning’s excellent regression capabilities, it has recently become the dominating technique for this assignment. Meanwhile, the development of the transformer, a novel deep learning architecture for natural language processing, has provided researchers with new insights. In this letter, we seek to extend transformer’s excellent mechanisms to pixel-level fusion challenges. We designed a parallel convolutional neural network structure for learning both the regions of interest from the LRMS images and the residuals required for regression to HRMS images. Then, in our proposed pixelwise attention constraint (PAC) module, the residuals will be changed utilizing the learned region of interest. In addition, we presented a novel multireceptive-field attention block (MRFAB) to frame our network. Experiments on two datasets also show that our work is better than the mainstream algorithms at both indicators and visualization.

MD³Net: Integrating Model-Driven and Data-Driven Approaches for Pansharpening

Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multi-modal fusion task, in this paper, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD³Net), which combines model-driven and data-driven approaches. The architecture design of MD³Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input/output predictable. In order to further explore the powerful learning ability of deep learning based approaches, we introduce the deep prior into the MD³Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared to a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet..

Spatial and Spectral Anisotropic Tensor Total Variation-Driven Adaptive Pansharpening

This letter proposed a spatial and spectral anisotropic tensor total variation (SSATTV) driven adaptive pansharpening model for the fusion of low-resolution (LR) multispectral (MS) and panchromatic (Pan) images to the high-resolution (HR) MS images. Except for the local spectral consistency constraint-based fidelity term between HR and LR MS used for spectral preservation, the proposed model reformulated the adaptive linear constraint-based fidelity term between Pan and HR MS with the tensor-representation modeling, and particularly proposed a SSATTV prior term which imposed the spatial anisotropic TV prior between HR MS and Pan for spatial gradient feature preservation, and the spectral TV sparsity prior on HR MS for further spectral feature preservation. Furthermore, the proposed model was efficiently solved via the alternating direction method of multipliers (ADMM) scheme. Specifically, the experiments validated the superiority of proposed SSATTV method.

LDP-Net: An Unsupervised Pansharpening Network Based on Learnable Degradation Processes

Pansharpening in remote sensing image aims at acquiring a high-resolution multispectral (HRMS) image directly by fusing a low-resolution multispectral (LRMS) image with a panchromatic (PAN) image. The main concern is how to effectively combine the rich spectral information of LRMS image with the abundant spatial information of PAN image. Recently, many methods based on deep learning have been proposed for the pansharpening task. However, these methods usually have two main drawbacks: 1) requiring HRMS for supervised learning; and 2) simply ignoring the latent relation between the MS and PAN image and fusing them directly. To solve these problems, we propose a novel unsupervised network based on learnable degradation processes, dubbed as LDP-Net. A reblurring block and a graying block are designed to learn the corresponding degradation processes, respectively. In addition, a novel hybrid loss function is proposed to constrain both spatial and spectral consistency between the pansharpened image and the PAN and LRMS images at different resolutions. Experiments on GaoFen-2, Worldview-2 and Worldview-3 images demonstrate that our proposed LDP-Net can fuse PAN and LRMS images effectively without the help of HRMS samples, achieving promising performance in terms of both qualitative visual effects and quantitative metrics.

Notice of Removal: P3Net: Pansharpening via Pyramidal Detail Injection With Deep Physical Constraints

Pansharpening is an image fusion process aiming to generate high-resolution multispectral (HRMS) images from a pair of low-resolution multispectral (LRMS) images and a high-resolution PAN image. It is a fundamental and significant task for the widespread use of remote sensing images. This paper proposes a new residual learning-based multispectral pansharpening network constrained by two deep physical models, collectively termed as P3Net. It mainly consists of the mainstream PDFNet and the other two auxiliary physical models, M2PNet and H2LNet. Unlike the existing methods of processing only one image scale, the proposed PDFNet fully extracts the spatial details from the multi-level image pyramid with decreasing spatial scales. Then, the spatial information is injected into the upsampled LRMS image. Since the pan-sharpened result should be consistent with the observed inputs under the physics models, we learn deep pansharpening physics models to reflect the inverse relationships. In detail, we propose the lightweight M2PNet and H2LNet to represent the latent non-linear mappings from the HRMS image to the panchromatic (PAN) image and the LRMS image, respectively. The two pre-trained physics models are frozen and guide the training of the PDFNet, so as to drive clear physical interpretability and further suppress the spectral and spatial distortions. The comparative experiments with the existing state-of-the-art pansharpening methods on QuickBird, GaoFen, and WorldView test sets demonstrate the superiority of the proposed method in terms of both quantitative metrics and subjective visual effects. The codes are available at https://github.com/KSJhon/PyramidPanWithPhysics.

Pansharpening via Triplet Attention Network With Information Interaction

Pansharpening aims to obtain high spatial resolution multispectral (MS) images by fusing the spatial and spectral information in low spatial resolution (LR) MS and panchromatic (PAN) images. Recently, deep neural network (DNN) based pansharpening methods have been advanced extensively. Although most DNN-based methods show good performance, it is difficult for them to preserve the spatial details in the fused image. In this article, we propose a new pansharpening method based on a triplet attention network with information interaction to efficiently enhance the spatial and spectral information in the fused image. First, different attention mechanisms are designed to model the spatial and spectral feature properties in LR MS and PAN images. Then, the complementarity among different feature maps is enhanced by information interaction, which promotes the compatibility of features from subnetworks. Finally, we utilize a graph attention module to capture the similarity within feature maps. According to the graph, the informative feature maps are selected to provide more details for the reconstruction of the fused image. Extensive experiments on QuickBird and GeoEye-1 satellite datasets show that the proposed method can produce competitive fused images when compared with some state-of-the-art methods.

Effective Pan-Sharpening by Multiscale Invertible Neural Network and Heterogeneous Task Distilling

As recognized, the ground truth multi-spectral (MS) images possess the complementary information (e.g., high-frequency component) of low-resolution (LR) MS images, which can be considered as privileged information to alleviate the spectral distortion and insufficient spatial texture enhancement. Since existing supervised pan-sharpening methods only utilize the ground truth MS image to supervise the network training, its potential value has not been fully explored. To accomplish this, we propose a heterogeneous knowledge-distilling pan-sharpening framework that distills pan-sharpening by imitating the ground truth reconstruction task in both the feature space and network output. In our work, the teacher network performs as a variational auto-encoder to extract effective features of the ground truth MS. The student network, acting as pan-sharpening, is trained by the assistance of the teacher network with the process-oriented feature imitation learning. Moreover, we design a customized information-lossless multi-scale invertible neural module to effectively fuse LR-MS and panchromatic (PAN) images, producing expected pan-sharpened results. To reduce the artifacts generated by the knowledge distillation process, a knowledge-driven refinement sub-network is further devised according to the pan-sharpening imaging model. Extensive experimental results on different satellite datasets validate that the proposed network outperforms the state-of-the-art methods both visually and quantitatively. The source code will be released at https://github.com/manman1995/pansharpening.

ArbRPN: A Bidirectional Recurrent Pansharpening Network for Multispectral Images With Arbitrary Numbers of Bands

Although the performance of pansharpening has been significantly improved by advanced deep-learning (DL) technologies in recent years, most DL-based methods fail to process multispectral (MS) images with arbitrary numbers of bands by a single model. Consequently, it is inevitable to train separate models for MS images with different numbers of bands, which is time- and storage-consuming as well as inefficient in practice. To tackle the above problem, we propose a bidirectional recurrent pansharpening network (named ArbRPN) for MS images with arbitrary numbers of bands. Our ArbRPN can dynamically reconstruct high-resolution (HR) MS images with different numbers of bands by adaptively changing the number of recurrence to the number of bands of the low-resolution (LR) MS images. Leveraging on the ability of the ArbRPN to process MS images with any number of bands, one can even customize the bands to be pansharpened. Moreover, to achieve superior performance, spectral discrepancy and dependence are considered in the ArbRPN. Details from the panchromatic (PAN) image are adaptively injected into the fused product according to the captured spectral dependence. Furthermore, training strategies of existing DL-based pansharpening methods can only group MS images with a constant number of bands into mini-batches. Therefore, we present a mask-based training method (called mask-training) to solve this problem. Benefiting from the mask-training, our ArbRPN can achieve superior performance and robustness during pansharpening. Extensive experiments show the superior performance of our ArbRPN with respect to the state-of-the-art (SOTA) methods applied to MS images with different numbers of bands. The code of our ArbRPN is available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Lihui-Chen/ArbRPN.git</uri> .

Detail Injection-Based Convolutional Auto-Encoder for Pansharpening

The purpose of pansharpening is to generate high-resolution multispectral (MS) images using both low-resolution MS images and high-resolution panchromatic images. Traditional remote sensing image fusion algorithms can be simplified to a unified detail injection (Di) context that treats the injected MS details as panchromatic-detail and integration with injection gain. The injected details are developed from traditional fusion strategies with clear physical interpretation and facilitate fast convergence of deep learning models for high-quality image fusion. The excellent ability of convolutional autoencoder (CAE) networks to retain image information enables its application to remote sensing image fusion. In this paper, a fusion method Di-based CAE (DiCAE) based on Di and CAE is proposed. DiCAE method is based on Di as the theoretical foundation and CAE network as the core of the algorithm. In addition, our method is evaluated through experiments on different satellite datasets, and the fusion results obtained by DiCAE have better objective evaluation metrics and better visual results compared to other state-of-the-art methods.

VP-Net: An Interpretable Deep Network for Variational Pansharpening

In this study, we propose an interpretable deep network for variational pansharpening (VP), named VP-Net. Different from traditional priors using linear operators, such as the gradient, we construct a prior based on the similarity between panchromatic (PAN) and high-resolution multispectral (HRMS) images by a nonlinear operator that can be learned through a deep network. Considering the spectral difference of various satellite multispectral (MS) imaging platforms, we specifically seek the aforementioned similarity from the PAN image and the intensity of the HRMS image to reduce the spectral distortion. Based on this prior, we propose a novel VP model by further incorporating a data fidelity term from the low-resolution MS image. Specifically, we build the VP-Net by unrolling the variable splitting method for an optimal solution to this model. Consequently, all modules in VP-Net have clear physical meanings and strong generalization capabilities. Meanwhile, all parameters and the aforementioned nonlinear operator are learned in VP-Net, avoiding the difficulty of selecting optimal handcrafted parameters in traditional methods. Therefore, VP-Net not only achieves an optimal balance between spatial and spectral qualities but also has a strong generalization capability across different types of training and testing data. In the experiment, we first demonstrate the superiority of the proposed method over the current state of the arts in terms of both visual effect and quantitative analysis on different satellite datasets. Moreover, we carry out an normalized difference vegetation index (NDVI) experiment to demonstrate its potential in remote sensing.

Gradient Guided Pyramidal Convolution Residual Network with Interactive Connections for Pan-sharpening

ABSTRACT Convolutional neural networks (CNNs) have played a predominant role in the field of remote sensing over the last few years. As a significant branch of remote sensing image processing, pan-sharpening technique is to produce a high-resolution multi-spectral (HRMS) image based on a low-resolution multi-spectral (LRMS) image and a high-resolution (HR) panchromatic (PAN) image. Benefiting from the inherently powerful representation ability, deep-learning-based methods have also achieved promising and favourable performance in pan-sharpening community. However, these methods don’t take advantage of the gradient characteristic which contains abundant structure information to guide the pan-sharpening process, failing to achieve the desired spatial preservation. In this paper, we propose a gradient-guided pyramidal convolution residual network with interactive connections (GGPCRN) to relieve the above issue. Specifically, besides the indispensable reconstruction branch, an auxiliary gradient branch providing additional structure information is built to guide the recovery process. Moreover, we introduce pyramidal convolution containing a series of filters with varying depth and size into our network to capture different scales of details for better performance. To further enhance the guidance of gradient maps, two measures are taken. On the one hand, interactive connections are proposed to transfer the mutual effect between the reconstruction branch and gradient branch. On the other hand, we incorporate a mild gradient loss to force a second-order restraint on the pan-sharpened images, making the network concentrate more on structure preservation. Both reduced-resolution and full-resolution experiments suggest that our GGPCRN performs favourably against other methods in terms of quantitative evaluations and visual improvements.

Pan-sharpening for compressed remote sensing images

Pan-sharpening is an effective method to obtain high-resolution multispectral (MS) images by fusing panchromatic (PAN) images with fine spatial structure and low-resolution MS images with rich spectral information. Many methods have been recently developed based on convolutional neural networks (CNNs) for pan-sharpening, which try to achieve better fusion performance using a large amount of training data. The huge demand for data of CNN-based methods results in huge pressure on transmission and storage conditions. Especially, PAN images occupy the main amount of data space because of whose spatial resolution is usually four times that of the corresponding MS images, which limits the application of pan-sharpening. We propose a CNN-based pan-sharpening method for compressed remote sensing to overcome this limitation, which can achieve pan-sharpening based on compressed data while eliminating compression artifacts. Specifically, we use compressed PAN images as the input and remove the compression artifacts, then fuse the PAN and MS images to achieve pan-sharpening. This method can greatly reduce the data capacity by data compression and ensure the pan-sharpening performance. Compared with the widely used state-of-the-art pan-sharpening approaches in a comprehensive evaluation, our method can obtain more promising results on compressed remote sensing images, and its superiority is thus demonstrated.

A Distributed Fusion Framework of Multispectral and Panchromatic Images Based on Residual Network

Remote sensing images have been widely applied in various industries; nevertheless, the resolution of such images is relatively low. Panchromatic sharpening (pan-sharpening) is a research focus in the image fusion domain of remote sensing. Pan-sharpening is used to generate high-resolution multispectral (HRMS) images making full use of low-resolution multispectral (LRMS) images and panchromatic (PAN) images. Traditional pan-sharpening has the problems of spectral distortion, ringing effect, and low resolution. The convolutional neural network (CNN) is gradually applied to pan-sharpening. Aiming at the aforementioned problems, we propose a distributed fusion framework based on residual CNN (RCNN), namely, RDFNet, which realizes the data fusion of three channels. It can make the most of the spectral information and spatial information of LRMS and PAN images. The proposed fusion network employs a distributed fusion architecture to make the best of the fusion outcome of the previous step in the fusion channel, so that the subsequent fusion acquires much more spectral and spatial information. Moreover, two feature extraction channels are used to extract the features of MS and PAN images respectively, using the residual module, and features of different scales are used for the fusion channel. In this way, spectral distortion and spatial information loss are reduced. Employing data from four different satellites to compare the proposed RDFNet, the results of the experiment show that the proposed RDFNet has superior performance in improving spatial resolution and preserving spectral information, and has good robustness and generalization in improving the fusion quality.

Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network

Real-time monitoring and surveillance play an important role in the field of remote sensing, where multi-spectral (MS) images with high spatial resolution are widely desired for better analysis. However, high-resolution MS images cannot be directly obtained due to the limitations of sensors and bandwidth. As an essential way to alleviate this problem, pan-sharpening aims at fusing the complementary information of a low-resolution MS image and a high-resolution panchromatic (PAN) image to reconstruct a high-resolution MS image. Most previous deep-learning based methods can meet the real-time requirements with the help of graphics processing unit (GPU). However, they don’t fully exploit the favorable hierarchical information, sparing huge room for performance improvement. In this paper, to meet the requirement of real-time implementation and achieve more effective performance simultaneously, we propose a multi-scale fusion network (MSFN) to make full use of hierarchical complementary features of PAN and MS images. Specifically, we introduce an encoder–decoder structure and coarse-to-fine strategy to effectively extract multi-scale features of PAN and MS images, separately. Meanwhile, an information pool is adopted to preserve primitive information. Then a multi-scale feature fusion module is applied to fuse multi-scale features from the decoder and information pool. Finally, the fused features are utilized to reconstruct the high-resolution MS image. Extensive experiments demonstrate that our proposed method achieves favorable performance against other methods in terms of quantitative metrics and visual quality. Besides, the results on running time indicate that our method can achieve real-time performance.

A primal-dual method for total-variation-based pansharpening

ABSTRACT This paper presents a variational framework to enhance the spatial details of the low-resolution (LR) multispectral (MS) image by the rich spatial information obtained from the panchromatic (Pan) image. The target high-resolution (HR) MS image is estimated through an inverse super-resolution problem, where the LR MS and Pan images are the observations. The LR MS image is modelled by the decimation of the target HR MS image which takes into account the modulation transfer function (MTF) of the MS sensor. In addition, the Pan image is described as a linear combination of the bands of the target HR MS image. A variational pansharpening model is defined according to the image observation models and the total variation (TV) regularization. The target HR MS image is obtained by optimizing the variational model using an efficient primal-dual algorithm in the Euclidean setting. Compared to the other variational pansharpening algorithms adopting the vector representation, the proposed algorithm solves the pansharpening problem by a primal-dual algorithm in the Euclidean setting, resulting in a highly efficient and less complex algorithm. The result of comparing the proposed algorithm with a number of state-of-the-art pansharpening methods demonstrates that the proposed algorithm is visually and quantitatively able to produce much better results. Moreover, the proposed algorithm has several advantages such as higher accuracy in preserving small objects and sharp features, faster convergence, and lower memory requirements over the existing variational pansharpening methods.

Second-Order Total Generalized Variation Regularization for Pansharpening

Pansharpening can be regarded as an inverse problem, where a high-resolution multispectral (MS) image is estimated given a low-resolution MS image and a panchromatic (Pan) image. Considering the effectiveness of total generalized variation (TGV) to regularize ill-posed inverse problems, this letter proposes a variational model in accordance with the observation model based on the satellite imaging system and the second-order TGV. Furthermore, a primal–dual algorithm is adopted to resolve the variational model by splitting it into a sequence of simpler sub-problems, which leads to a more efficient algorithm compared to the other variational methods. In addition, the exploitation of TGV in the variational model allows the presence of the very fine details of the Pan image in the sharpened MS image whereas it is beyond the capabilities of the total variation (TV)-based models. The proposed algorithm is compared with some recent classical and variational pansharpening methods by experiments on two data sets at full and reduced resolution.

Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening

The fusion of high spatial resolution panchromatic (PAN) data with simultaneously acquired multispectral (MS) data with the lower spatial resolution is a hot topic, which is often called pansharpening. In this article, we exploit the combination of machine learning techniques and fusion schemes introduced to address the pansharpening problem. In particular, deep convolutional neural networks (DCNNs) are proposed to solve this issue. The latter is combined first with the traditional component substitution and multiresolution analysis fusion schemes in order to estimate the nonlinear injection models that rule the combination of the upsampled low-resolution MS image with the extracted details exploiting the two philosophies. Furthermore, inspired by these two approaches, we also developed another DCNN for pansharpening. This is fed by the direct difference between the PAN image and the upsampled low-resolution MS image. Extensive experiments conducted both at reduced and full resolutions demonstrate that this latter convolutional neural network outperforms both the other detail injection-based proposals and several state-of-the-art pansharpening methods.

Sparse representation-based detail-injection method for pan-sharpening

Pan-sharpening is an important image preprocessing technique for remote sensing that aims to enhance spatial resolution of multispectral (MS) images under the guidance of panchromatic (PAN) image while preserving spectral properties. The existing pan-sharpening methods usually adopt the globally consistent detail-injection models, neglecting the detail differences between spectral channels, which leads to imprecise spatial details and distorted spectral properties of pan-sharpening results. We propose a sparse representation-based detail-injection model for pan-sharpening that utilizes the structure similarity and detail differences between PAN and low-resolution multispectral (LRM) images at each channel, to improve the performance of pan-sharpening. Specifically, to better express the inherent detail properties of the MS image, the overcomplete dictionary of each channel is constructed from synthesized high-resolution multispectral (HRM) images. Moreover, the most proposed methods require that the spectral responses of the PAN image and the MS image cover the same wavelength range; nevertheless, most sensors cannot match this condition. To address this problem, we propose constructing coupled low-resolution and high-resolution dictionaries from LRM and synthesized HRM images so that the structure similarities can be used for detail injection. The qualitative and quantitative experimental results on various data sets demonstrate the superiority of our proposed method over the state-of-the-art methods.

Combination of multifeature extraction and learning-based pooling for quality assessment of pan-sharpened remote sensing image

Fusion results of low-resolution multispectral (LRMS) images and high-resolution panchromatic (Pan) images, also called pan-sharpened images, are always difficult to evaluate due to the lack of high-resolution multispectral (HRMS) images and the complexity of the fusion process. By taking spectral information of LRMS images and spatial structural information of Pan images as references, we extract the saturation map and luminance value as spectral features, and construct the optimal contrast map and structure similarity map as spatial features to compute the four indices between the original LRMS, Pan images, and the fused result: saturation similarity, luminance consistency, contrast similarity, and structure similarity to describe distortions from different aspects. Then, we feed the four indices into an extreme learning machine to train a nonlinear pooling strategy, and finally a multifeature and learning-based model is constructed for fusion image quality assessment. Comparisons with state-of-the-art image quality assessment metrics show that the proposed metric gains a much higher consistency with subjective opinions while needing no reference HRMS images.