Panchromatic Research Articles

Detail Information Prior Net for Remote Sensing Image Pansharpening

Pansharpening, which fuses the panchromatic (PAN) band with multispectral (MS) bands to obtain an MS image with spatial resolution of the PAN images, has been a popular topic in remote sensing applications in recent years. Although the deep-learning-based pansharpening algorithm has achieved better performance than traditional methods, the fusion extracts insufficient spatial information from a PAN image, producing low-quality pansharpened images. To address this problem, this paper proposes a novel progressive PAN-injected fusion method based on superresolution (SR). The network extracts the detail features of a PAN image by using two-stream PAN input; uses a feature fusion unit (FFU) to gradually inject low-frequency PAN features, with high-frequency PAN features added after subpixel convolution; uses a plain autoencoder to inject the extracted PAN features; and applies a structural similarity index measure (SSIM) loss to focus on the structural quality. Experiments performed on different datasets indicate that the proposed method outperforms several state-of-the-art pansharpening methods in both visual appearance and objective indexes, and the SSIM loss can help improve the pansharpened quality on the original dataset.

A Distributed Fusion Framework of Multispectral and Panchromatic Images Based on Residual Network

Remote sensing images have been widely applied in various industries; nevertheless, the resolution of such images is relatively low. Panchromatic sharpening (pan-sharpening) is a research focus in the image fusion domain of remote sensing. Pan-sharpening is used to generate high-resolution multispectral (HRMS) images making full use of low-resolution multispectral (LRMS) images and panchromatic (PAN) images. Traditional pan-sharpening has the problems of spectral distortion, ringing effect, and low resolution. The convolutional neural network (CNN) is gradually applied to pan-sharpening. Aiming at the aforementioned problems, we propose a distributed fusion framework based on residual CNN (RCNN), namely, RDFNet, which realizes the data fusion of three channels. It can make the most of the spectral information and spatial information of LRMS and PAN images. The proposed fusion network employs a distributed fusion architecture to make the best of the fusion outcome of the previous step in the fusion channel, so that the subsequent fusion acquires much more spectral and spatial information. Moreover, two feature extraction channels are used to extract the features of MS and PAN images respectively, using the residual module, and features of different scales are used for the fusion channel. In this way, spectral distortion and spatial information loss are reduced. Employing data from four different satellites to compare the proposed RDFNet, the results of the experiment show that the proposed RDFNet has superior performance in improving spatial resolution and preserving spectral information, and has good robustness and generalization in improving the fusion quality.

MDCwFB: A Multilevel Dense Connection Network with Feedback Connections for Pansharpening

In most practical applications of remote sensing images, high-resolution multispectral images are needed. Pansharpening aims to generate high-resolution multispectral (MS) images from the input of high spatial resolution single-band panchromatic (PAN) images and low spatial resolution multispectral images. Inspired by the remarkable results of other researchers in pansharpening based on deep learning, we propose a multilevel dense connection network with a feedback connection. Our network consists of four parts. The first part consists of two identical subnetworks to extract features from PAN and MS images. The second part is a multilevel feature fusion and recovery network, which is used to fuse images in the feature domain and to encode and decode features at different levels so that the network can fully capture different levels of information. The third part is a continuous feedback operation, which refines shallow features by feedback. The fourth part is an image reconstruction network. High-quality images are recovered by making full use of multistage decoding features through dense connections. Experiments on different satellite datasets show that our proposed method is superior to existing methods, through subjective visual evaluation and objective evaluation indicators. Compared with the results of other models, our results achieve significant gains on the multiple objective index values used to measure the spectral quality and spatial details of the generated image, namely spectral angle mapper (SAM), relative global dimensional synthesis error (ERGAS), and structural similarity (SSIM).

PSMD-Net: A Novel Pan-Sharpening Method Based on a Multiscale Dense Network

Pan sharpening is used to fuse a low-resolution multispectral (MS) image and a high-resolution panchromatic (PAN) image to obtain a high-resolution MS image. This article proposes PSMD-Net, an end-to-end pan-sharpening method based on a multi-scale dense network. A shallow feature extraction layer (SFEL) extracts the shallow features from the original images, and these are used as an input to a global dense feature fusion (GDFF) network to learn the global features for image reconstruction. A multiscale dense block (MDB) is designed to fully extract the spatial and spectral information from the shallow features in the GDFF network. In the proposed network, multiple MDBs are stacked to extract rich, multi-scale dense hierarchical features, and a global dense connection (GDC) is designed to allow direct connections from the state of the current MDB to all subsequent MDBs to extract more advanced features. The extracted hierarchical features are sent to the global feature fusion layer (GFFL) to adaptively learn the global features for image reconstruction. Finally, global residual learning (GRL) is adopted to force the network to pay more attention to the changing part of the image. We perform experiments on simulated and real data from WorldView-2 and WorldView-3 satellites. Visual and quantitative assessment results demonstrate that PSMD-Net yields higher-resolution fusion images than the state-of-the-art methods.

Two Stages Pan-Sharpening Details Injection Approach Based on Very Deep Residual Networks

Pan-sharpening is a fusion task, which aims to combine a low spatial resolution multispectral (MS) image with a high spatial resolution single band panchromatic (PAN) image to produce a high spatial and spectral Pan-sharpened image. The success of a Pan-sharpening technique depends on its ability to boost the spatial quality of the MS image while preserving its spectral feature. To this end, we propose in this article a new two-stage detail injection approach allowing to reconstruct fine structures based on convolutional neural networks (CNNs). First, generalized Laplacian pyramid gain injections CNN is performed to estimate the optimal values of the injection gains for each MS band to inject spatial details extracted from the PAN image. Next, the result is enhanced by injecting the details missing using the power of deep residual learning. The quantitative and qualitative results on data sets from different satellites show that the proposed approach can achieve higher performances in both spatial and spectral qualities compared to the state of the art as well as the new CNN-based methods.

Pansharpening of WorldView-2 Data via Graph Regularized Sparse Coding and Adaptive Coupled Dictionary.

The spectral mismatch between a multispectral (MS) image and its corresponding panchromatic (PAN) image affects the pansharpening quality, especially for WorldView-2 data. To handle this problem, a pansharpening method based on graph regularized sparse coding (GRSC) and adaptive coupled dictionary is proposed in this paper. Firstly, the pansharpening process is divided into three tasks according to the degree of correlation among the MS and PAN channels and the relative spectral response of WorldView-2 sensor. Then, for each task, the image patch set from the MS channels is clustered into several subsets, and the sparse representation of each subset is estimated through the GRSC algorithm. Besides, an adaptive coupled dictionary pair for each task is constructed to effectively represent the subsets. Finally, the high-resolution image subsets for each task are obtained by multiplying the estimated sparse coefficient matrix by the corresponding dictionary. A variety of experiments are conducted on the WorldView-2 data, and the experimental results demonstrate that the proposed method achieves better performance than the existing pansharpening algorithms in both subjective analysis and objective evaluation.

Fusion of Hyperspectral and Panchromatic Images Using Generative Adversarial Network and Image Segmentation

Hyperspectral (HS) image fusion aims at integrating a panchromatic (PAN) image and an HS image, featuring the fused image with the spatial quality of the former and the spectral diversity of the latter. The classic fusion algorithm generally includes three consecutive procedures that are upsampling, detail extraction, and detail injection. In this article, we propose an HS and PAN image fusion method based on generative adversarial network and local estimation of injection gain. Instead of upsampling the HS image by classical interpolation techniques, a generative adversarial super-resolution network (GASN) is designed to obtain the interpolated HS image in the fusion framework. GASN establishes a spectral-information-based discriminator to conduct adversarial learning with the generator, so as to preserve the spectral information of the low-resolution HS image. An image segmentation-based injection gain estimation (ISGE) algorithm is subsequently proposed for HS and PAN images fusion. The injection gain is estimated over image segments obtained by a binary partition tree approach to improve the fusion performance. The proposed GASN and ISGE are implemented into two credible global estimation pansharpening methods, and experimental results prove the performance improvement of the proposed method. The proposed method is also compared with existing state-of-the-art methods, and experiments on several public databases demonstrate that the proposed method is competitive or superior to the state-of-the-art fusion methods.

Pan-sharpening based on multi-objective decision for multi-band remote sensing images

Pan-sharpening applies details injection to fuse a multispectral (MS) image with its corresponding panchromatic (PAN) image to produce a synthetic image. Theoretically, the synthetic image's spectral resolution should equal that of the MS image and its spatial resolution is the same as that of the PAN image. However, for existing pan-sharpening methods, the trade-off between the spectral and intensity information in the process of details injection is insufficient, resulting in spatial or spectral distortion of the fused image. In this paper we propose a novel pan-sharpening algorithm based on multi-objective decision for multi-band remote sensing images to improve the quality of the fused image. The proposed method focuses on developing a parametric model from a multi-objective perspective to simultaneously maximize the quality of all the pixels in the fused image. We introduce a details injection approach to enhance the edge and texture of the MS image. We design an efficient spectral fidelity fusion model based on the injected details using spectral modulation to pan-sharpen the MS image. We provide an algorithm based on multi-objective decision to solve this model. The main advantage of the proposed method is that it can provide effective spectral modulation to eliminate the adverse effects of details injection. We conduct experiments on simulated and real satellite image datasets to evaluate the proposed method. The results show that our method achieves superior performance to other state-of-the-art methods.

Pan-Sharpening Based on Panchromatic Image Spectral Learning Using WorldView-2

In order to adequately preserve the spatial information of panchromatic (PAN) image and the spectral information of multispectral (MS) image, this study proposes a method to learn the spectral information of MS based on the PAN image. In the spectral learning model, the PAN is degraded to the spatial resolution of the MS. Then, a deep convolutional neural network (CNN) is adopted to learn the spectral information between the degraded PAN and the original MS. In addition, the spectral evaluation index—spectral angle mapper (SAM) is used for controlling the spectral losses. Finally, the high spatial resolution MS (HMS) can be obtained through the trained model by using the original PAN as the test input. Besides, seven representative pan-sharpening algorithms and seven widely recognized objective fusion metrics are used to compare and evaluate the performance on the WorldView-2 experimental data, respectively. The results show that the proposed method achieves the purpose of pan-sharpening well, especially maintaining the optimal spectral information.

Hybrid Attention Based Residual Network for Pansharpening

Pansharpening aims at fusing the rich spectral information of multispectral (MS) images and the spatial details of panchromatic (PAN) images to generate a fused image with both high resolutions. In general, the existing pansharpening methods suffer from the problems of spectral distortion and lack of spatial detail information, which might prevent the accuracy computation for ground object identification. To alleviate these problems, we propose a Hybrid Attention mechanism-based Residual Neural Network (HARNN). In the proposed network, we develop an encoder attention module in the feature extraction part to better utilize the spectral and spatial features of MS and PAN images. Furthermore, the fusion attention module is designed to alleviate spectral distortion and improve contour details of the fused image. A series of ablation and contrast experiments are conducted on GF-1 and GF-2 datasets. The fusion results with less distorted pixels and more spatial details demonstrate that HARNN can implement the pansharpening task effectively, which outperforms the state-of-the-art algorithms.

Laplacian Pyramid Dense Network for Hyperspectral Pansharpening

Hyperspectral (HS) pansharpening aims to create a pansharpened image that integrates the spatial details of the panchromatic (PAN) image and the spectral content of the HS image. In this article, we present a deep convolutional network within the mature Gaussian–Laplacian pyramid for pansharpening (LPPNet). The overall structure of LPPNet is a cascade of the Laplacian pyramid dense network with a similar structure at each pyramid level. Following the general idea of multiresolution analysis (MRA), the subband residuals of the desired HS images are extracted from the PAN image and injected into the upsampled HS image to reconstruct the high-resolution HS images level by level. Applying the mature Laplace pyramid decomposition technique to the convolution neural network (CNN) can simplify the pansharpening problem into several pyramid-level learning problems so that the pansharpening problem can be solved with a shallow CNN with fewer parameters. Specifically, the Laplacian pyramid technology is used to decompose the image into different levels that can differentiate large- and small-scale details, and each level is handled by a spatial subnetwork in a divide-and-conquer way to make the network more efficient. Experimental results show that the proposed LPPNet method performs favorably against some state-of-the-art pansharpening methods in terms of objective indexes and subjective visual appearance.

Fusion of Multispectral and Panchromatic Images by Integrating Standard PCA with Rotated Wavelet Transform

Many pansharpening algorithms are based on the principle of extracting spatial details from panchromatic (PAN) images and injecting them into multispectral (MS) images. In this paper, we present two fusion approach based on same principle by integrating standard principle component analysis (PCA) with decimated and undecimated rotated wavelet transform. When decimated/subsampled rotated wavelet transform (SSRWT) is used for fusion of MS and PAN images, three visual artifacts get introduced in the fused image namely color distortion, shifting effect and shift distortion. To eliminate color distortion, SSRWT is integrated with standard PCA, i.e., PCA–SSRWT. Color distortion is significantly mitigated, but shifting effect and shift distortion persist in the fused image of PCA–SSRWT. After employing undecimated/nonsubsampled rotated wavelet transform (NSRWT), shifting effect and shift distortion get eliminated with minimum color distortion. However, fused image as a result of NSRWT is spectrally high but spatially low. In order to improve spatial quality and remove visual artifacts observed in SSRWT and PCA–SSRWT, NSRWT is integrated with standard PCA, i.e., PCA–NSRWT. Visual and quantitative analysis is carried out to validate the quality of fused image for all the algorithms. Visual interpretation suggests that fused image obtained using PCA–NSRWT is superior to fused images of SSRWT, PCA and NSRWT. The overall quantitative analysis manifests that the PCA–NSRWT is consistent with visual interpretation and performs better than state-of-the-art methods. PCA–NSRWT not only removes visual artifacts but also improves spectral and spatial quality of the fused image compared to individual PCA, SSRWT, NSRWT and PCA–SSRWT. Based on visual and quantitative analysis, it is observed that PCA works better with undecimated compared to decimated rotated wavelet transform for fusion.

A Two-Stage PAN-Sharpening Algorithm Based on Sparse Representation for Spectral Distortion Reduction

Inspite of technological advancement, inherent processing capability of current age sensors limits the desired details in the acquired image for variety of remote sensing applications. Pan-sharpening is a prominent scheme to integrate the essential spatial details inferred from panchromatic (PAN) image and the desired spectral information of multispectral (MS) image. This paper presents an effective two-stage pan-sharpening method to produce high resolution multispectral (HRMS) image. The proposed method is based on the premise that the HRMS image can be formulated as an amalgam of spectral and spatial components. The spectral components are estimated by processing the interpolated MS image with a filter approximated with modulation transfer function (MTF) of the sensor. Sparse representation theory is adapted to construct the spatial components. The high-frequency details extracted from the PAN image and its low resolution variant are utilized to construct dual dictionaries. The dictionaries are jointly learned by an efficient training algorithm to enhance the adaptability. The hypothesis of sparse coefficients invariance over scales is also incorporated to reckon the appropriate spatial information. Further, an iterative filtering mechanism is developed to enhance the quality of fused image. Four distinct datasets generated from QuickBird, IKONOS, Pléiades and WorldView-2 sensors are used for experimentation. The comprehensive assessment at reduced-scale and full-scale persuade the effectiveness of proposed method in the retention of spectral information and intensification of the spatial details.

Remote sensing image fusion with multistream deep ResCNN

Remote sensing image pan-sharpening is the fusion of multispectral (MS) image and panchromatic (PAN) image into fused image with MS and high-resolution information features. In recent years, deep learning technology has been widely used in abounding fields such as text and image recognition. Inspired by this, we proposed a multistream deep ResCNN (RIFRCNN)-based fusion method for remote sensing images. First, we put forward automatic learning matching module (ALMM). The utilization of ALMM can realize dynamic optimization matching of MS and PAN images with different resolutions. Second, we build feature fusion residual blocks suitable for RIFRCN to optimize network parameters. Finally, we input MS image and PAN image with different resolution and complete the fusion task through ALMM, fusion, and reconstruction module. Experiments on WorldView and QuickBird datasets manifest that the proposed RIFRCNN can effectively fuse MS and PAN images, and the fusion images generated in the experiments of true and false color images show better results than the state-of-the-art technology.

Image Fusion-The Pioneering Technique for Real-Time Image Processing Applications

An image is a two-dimensional function that is expressed through spatial coordinates X, Y. At any pair of coordinates (x, y), the amplitude of a point is called the intensity of that pixel. Digital Image comprises a predictable number of components, each of which has a precise value at a given region. Those components are called pixels. Image Fusion is a phenomenon of transforming data from two or more images of a scenario into a single, more descriptive image taken than both of the input images, and is more appropriate for information processing. Image Fusion (IF) has been utilized in numerous application regions/areas. Remote Sensing Satellites (RSS) produce different images based on their sensory characteristics. Among those images, Panchromatic (PAN) and Multi-Spectral (MS) images are widely used in Satellite Image Fusion (SIF). The Image Fusion (IF) techniques are broadly classified as methods for the Spatial and Frequency domains. Wavelet Fusion Techniques (WFT) based on the Frequency-Domain (FD) are having applications in medical, space, and military applications. This literature delivers a study of some of the Image Fusion (IF) techniques. Remote Sensing Image (RSI) and Data Fusion (DF) seeks to merge the data acquired from sensors installed on satellites, airborne platforms, and ground-based sensors with specific spatial, spectral and temporal resolutions to produce merged data containing more accurate information than is found in each of the individual data sources.

MSSL: Hyperspectral and Panchromatic Images Fusion via Multiresolution Spatial–Spectral Feature Learning Networks

The fusion of hyperspectral (HS) and panchromatic (PAN) images aims to generate a fused HS image that combines spectral information of the HS image with spatial information of the PAN image. In this article, we propose a multiresolution spatial–spectral feature learning (MSSL) framework for fusing HS and PAN images. The proposed MSSL transforms the existing deep and complex network into several simple and shallow subnetworks to simplify the feature learning process. MSSL upsamples the HS image while downsamples the PAN image and designs multiresolution 3-D convolutional autoencoder (CAEs) networks with a spectral constraint to learn complete spatial–spectral features of the HS image. MSSL designs multiresolution 2-D CAEs with spatial constraint to extract spatial features of the PAN image, with a low computational cost. In order to effectively generate the pansharpened HS image with high spatial and spectral fidelity, a multiresolution residual network is presented to reconstruct the HS image from the extracted spatial–spectral features. Extensive experiments are conducted on three widely used remote sensing data sets in comparison with state-of-the-art HS image fusion methods, demonstrating the superiority of the proposed MSSL method. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Jiahuiqu/MSSL</uri> .

MSDRN: Pansharpening of Multispectral Images via Multi-Scale Deep Residual Network

In order to acquire a high resolution multispectral (HRMS) image with the same spectral resolution as multispectral (MS) image and the same spatial resolution as panchromatic (PAN) image, pansharpening, a typical and hot image fusion topic, has been well researched. Various pansharpening methods that are based on convolutional neural networks (CNN) with different architectures have been introduced by prior works. However, different scale information of the source images is not considered by these methods, which may lead to the loss of high-frequency details in the fused image. This paper proposes a pansharpening method of MS images via multi-scale deep residual network (MSDRN). The proposed method constructs a multi-level network to make better use of the scale information of the source images. Moreover, residual learning is introduced into the network to further improve the ability of feature extraction and simplify the learning process. A series of experiments are conducted on the QuickBird and GeoEye-1 datasets. Experimental results demonstrate that the MSDRN achieves a superior or competitive fusion performance to the state-of-the-art methods in both visual evaluation and quantitative evaluation.

Panchromatic Image Super-Resolution Via Self Attention-Augmented Wasserstein Generative Adversarial Network.

Panchromatic (PAN) images contain abundant spatial information that is useful for earth observation, but always suffer from low-resolution ( LR) due to the sensor limitation and large-scale view field. The current super-resolution (SR) methods based on traditional attention mechanism have shown remarkable advantages but remain imperfect to reconstruct the edge details of SR images. To address this problem, an improved SR model which involves the self-attention augmented Wasserstein generative adversarial network ( SAA-WGAN) is designed to dig out the reference information among multiple features for detail enhancement. We use an encoder-decoder network followed by a fully convolutional network (FCN) as the backbone to extract multi-scale features and reconstruct the High-resolution (HR) results. To exploit the relevance between multi-layer feature maps, we first integrate a convolutional block attention module (CBAM) into each skip-connection of the encoder-decoder subnet, generating weighted maps to enhance both channel-wise and spatial-wise feature representation automatically. Besides, considering that the HR results and LR inputs are highly similar in structure, yet cannot be fully reflected in traditional attention mechanism, we, therefore, designed a self augmented attention (SAA) module, where the attention weights are produced dynamically via a similarity function between hidden features; this design allows the network to flexibly adjust the fraction relevance among multi-layer features and keep the long-range inter information, which is helpful to preserve details. In addition, the pixel-wise loss is combined with perceptual and gradient loss to achieve comprehensive supervision. Experiments on benchmark datasets demonstrate that the proposed method outperforms other SR methods in terms of both objective evaluation and visual effect.

A pansharpened image quality assessment using segmentation procedure

ABSTRACT Pansharpening is an important way of integrating spatial and spectral information in the field of remote sensing. This field uses the complementary and redundant information between multispectral (MS) images and panchromatic (PAN) images to obtain high spectral and high spatial resolution images. Various pansharpening methods have been introduced so far, each one attempting to provide a pansharpened image with the least distortion and maximum preservation of spectral and spatial information. Due to the importance of this issue, there should be methods and indices to evaluate the performance of different pansharpening algorithms and assess the quality of pansharpened images. In this paper, a segmentation-based method for assessing the quality of fused images is proposed. The advantage of this approach over pixel-based methods is that the pixel-based methods consider the fused images as a set of separate pixels while segmentation can take into account useful spatial information such as neighbourhoods, textures, etc. In the proposed method, by using k-means clustering algorithm, the reference and pansharpened images are segmented into areas with similar spectral and spatial features and the corresponding segments of the images are compared. This method is tested on three real data sets acquired by Pleiades, GeoEye-1, and QuickBird sensors. Experimental results demonstrate the effectiveness of the proposed method in evaluation of the quality of fused images.

A Novel Adaptive Hybrid Fusion Network for Multiresolution Remote Sensing Images Classification

With the rapid development of earth observation technology, panchromatic (PAN) and multispectral (MS) images have also become easier to obtain. The multiresolution classification of PAN and MS images as a basic MS image analysis task has become a research hotspot. The main challenge in this field is how to process data and extract features to improve classification accuracy effectively. In this article, we design a novel adaptive hybrid fusion network (AHF-Net) for multiresolution remote sensing image classification. It includes two parts: data fusion and feature fusion. In the data fusion part, we propose an adaptive weighted intensity-hue-saturation (AWIHS) strategy, which can reduce the difference between MS and PAN images by adaptively adding each other’s unique information from the perspective of information sharing. In the feature fusion part, starting from the second-order correlation of features, we propose a correlation-based attention feature fusion (CAFF) module. It can improve the discrimination of fusion features by adaptively determining the fusion coefficient according to the importance of the input feature channel. Based on AWIHS and CAFF, inspired by the idea of feature pyramid, we combine the multilevel feature fusion and the dual-branch residual network as the backbone network of AHF-Net. By combining AWIHS and CAFF modules with the backbone network, our AHF-Net can effectively improve the classification accuracy of multiresolution remote sensing images. The effectiveness of the proposed algorithm has been verified on multiple data sets. Our code and model are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/1826133674/AHF-Net</uri> .