Panchromatic Research Articles

PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening

This article addresses the problem of remote sensing image pan-sharpening from the perspective of generative adversarial learning. We propose a novel deep neural network-based method named pansharpening GAN (PSGAN). To the best of our knowledge, this is one of the first attempts at producing high-quality pan-sharpened images with generative adversarial networks (GANs). The PSGAN consists of two components: a generative network (i.e., generator) and a discriminative network (i.e., discriminator). The generator is designed to accept panchromatic (PAN) and multispectral (MS) images as inputs and maps them to the desired high-resolution (HR) MS images, and the discriminator implements the adversarial training strategy for generating higher fidelity pan-sharpened images. In this article, we evaluate several architectures and designs, namely, two-stream input, stacking input, batch normalization layer, and attention mechanism to find the optimal solution for pan-sharpening. Extensive experiments on QuickBird, GaoFen-2, and WorldView-2 satellite images demonstrate that the proposed PSGANs not only are effective in generating high-quality HR MS images and superior to state-of-the-art methods but also generalize well to full-scale images.

Morphology-based spatial filtering for efficiency enhancement of remote sensing image fusion

Better visual interpretation and high-level feature extraction are always desired for remotely sensed image processing applications. The fulfillment is most prominent when a Multi-Spectral (MS) image fused with a PANchromatic (PAN) image for the same geographic location produces another MS image with added spatial resolution. Development of such fusion algorithms based on approaches like Component Substitution (CS) and Multi-Resolution Analysis (MRA) is continuously researched over the last few decades. In this paper, the authors propose the morphological operator-based image fusion algorithm featuring nonlinear decomposition. Also, the use of morphological operator based spatial filtering is successfully demonstrated for efficiency enhancement of the proposed and a few of the sophisticated image fusion algorithms based on CS, MRA, and state-of-the-art deep learning approach. The potential of the presented work is proved through reduced- and full-resolution assessment procedures utilizing two data sets acquired by WorldView-3 and WorldView-2 satellite sensors.

Object-Based Area-to-Point Regression Kriging for Pansharpening

Optical earth observation satellite sensors often provide a coarse spatial resolution (CR) multispectral (MS) image together with a fine spatial resolution (FR) panchromatic (PAN) image. Pansharpening is a technique applied to such satellite sensor images to generate an FR MS image by injecting spatial detail taken from the FR PAN image while simultaneously preserving the spectral information of MS image. Pansharpening methods are mostly applied on a per-pixel basis and use the PAN image to extract spatial detail. However, many land cover objects in FR satellite sensor images are not illustrated as independent pixels, but as many spatially aggregated pixels that contain important semantic information. In this article, an object-based pansharpening approach, termed object-based area-to-point regression kriging (OATPRK), is proposed. OATPRK aims to fuse the MS and PAN images at the object-based scale and, thus, takes advantage of both the unified spectral information within the CR MS images and the spatial detail of the FR PAN image. OATPRK is composed of three stages: image segmentation, object-based regression, and residual downscaling. Three data sets acquired from IKONOS and Worldview-2 and 11 benchmark pansharpening algorithms were used to provide a comprehensive assessment of the proposed OATPRK approach. In both the synthetic and real experiments, OATPRK produced the most superior pan-sharpened results in terms of visual and quantitative assessment. OATPRK is a new conceptual method that advances the pixel-level geostatistical pansharpening approach to the object level and provides more accurate pan-sharpened MS images.

Fusion of Panchromatic and Multispectral Images Using Multiscale Convolution Sparse Decomposition

In this article, we proposed a novel image fusion method based on multiscale convolution sparse decomposition (MCSD). A unified framework based on MCSD is first utilized to decompose panchromatic (PAN) image and the spatial component of upsampled low spatial resolution multispectral (LR MS) images, which can produce the corresponding low frequencies and feature maps. By combining convolution sparse decomposition with multiscale analysis, MCSD can efficiently approximate the spatial and spectral information in images. Next, a binary map generated from gradient information is utilized to integrate the low frequencies of LR MS and PAN images. For feature maps, the fusion gain for each pixel is calculated according to the similarity between the local patches from them. Finally, the fused image is reconstructed by the sum of fused low frequency and feature maps. Some experiments are conducted on QuickBird and GeoEye-1 satellite datasets. Compared with other methods, the proposed method performs better in visual and numerical evaluations.

An efficient and high-quality pansharpening model based on conditional random fields

Pansharpening fuses a low spatial resolution multi-spectral (MS) image with the corresponding panchromatic (PAN) image to obtain a high spatial resolution MS (HRMS) image. Traditional fusion methods may easily cause a spectral or spatial distortion when injecting details into an MS image. To preserve the spectral and spatial information, an efficient pansharpening model based on conditional random fields (CRFs) is proposed. With this model, a state feature function is designed to force the HRMS image filtered using a blur function to be consistent with the up-sampled MS image and retain the spectral fidelity. To obtain a proper blur function, a new filter-acquisition method is proposed for the unified CRF-based model. Meanwhile, a transition feature function is defined to enable the transition of HRMS pixels to follow the gradient of a PAN image and ensure the sharpness of the fused image. Considering the characteristics of the gradient domain, a total variation regularization is designed to make the gradient of the HRMS image sparse. Finally, the augmented Lagrangian function of the model is solved by employing an alternating direction method of the multipliers. Experiment results indicate that, compared with previous state-of-the-art pansharpening methods, the proposed method can achieve the best fusion results with high computational efficiency.

Spatial Sharpening of KOMPSAT-3A MIR Images Using Optimal Scaling Factor

This paper present efficient methods for merging KOMPSAT-3A (Korea Multi-Purpose Satellite) medium wave Infrared (MIR) and panchromatic (PAN) images. Spatial sharpening techniques have been developed to create an image with both high spatial and high spectral resolution by combining the desired qualities of a PAN image with high spatial and low spectral resolution and an MS/MIR image with low spatial and high spectral resolution. The proposed methods can extract an optimal scaling factor, and uses the tactics of appropriately controlling the balance between the spatial and spectral resolutions. KOMPSAT-3A PAN and MIR images were used to test and evaluate the performance of the proposed methods. A qualitative assessment were performed using the image quality index (Q4), the cross correlation index (CC) and the relative global dimensional synthesis error (Spectral/Spatial ERGAS). These tests indicate that the proposed methods preserve the spectral and spatial characteristics of the original MIR and PAN images. Visual analysis reveals that the spectral and spatial information derived from the proposed methods were well retained in the test images. A comparison of the results of the proposed methods with those obtained from applying existing ones such as the Multi Sensor Fusion (MSF) technique or the Guide Filter Based Fusion (GF) show the efficiency of the new fusion process to be superior to the one of the others. The results showed a significant improvement in fusion capability for KOMPSAT-3A MIR imagery.

A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening With Classical and Emerging Pansharpening Methods

Pansharpening refers to the fusion of a multispectral (MS) image and panchromatic (PAN) data aimed at generating an outcome with the same spatial resolution of the PAN data and the spectral resolution of the MS image. In the last 30 years, several approaches to deal with this issue have been proposed. However, the reproducibility of these methods is often limited, making the comparison with the state of the art hard to achieve. Thus, to fill this gap, we propose a new benchmark consisting of recent advances in MS pansharpening. In particular, optimized classical approaches [multiresolution analysis (MRA) and component substitution (CS)] are compared with methods belonging to the third generation of pansharpening, represented by variational optimization-based (VO) and machine learning (ML) techniques. The benchmark is tested on different scenarios (from urban to rural areas) acquired by different commercial sensors [i.e., IKONOS (IK), GeoEye-1 (GE-1), and WorldView-3 (WV-3)]. Both quantitative and qualitative assessments and the computational burden are analyzed in this article, and all of the implementations have been collected in a MATLAB toolbox that is made available to the community.

A variational pan-sharpening algorithm to enhance the spectral and spatial details

ABSTRACT Pan-sharpening is a remote sensing image fusion technique that generates a high-resolution multispectral (HRMS) image on combining a low resolution multispectral (MS) image and a panchromatic (PAN) image. In this paper, a new optimisation model is proposed for pan-sharpening. The proposed model consists of three terms: (i) a data synthesis fidelity term formulated on inferring the relationship between source MS image and fused image to preserve the spectral information, (ii) a total generalised variation-based prior term to inject the significant spatial details from PAN image to pan-sharpened image, and (iii) a spectral distortion reduction term that exploits the correlation between multispectral image bands. To solve the resultant convex optimisation problem, an efficient and convergence guaranteed operator splitting framework based on the alternating direction method of multipliers (ADMM) algorithm is formulated. Finally, the proposed model is experimentally validated using full-resolution and reduced-resolution data. The pan-sharpened outcomes exhibit the potential of the proposed method in enhancing the spatial and spectral quality.

Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening

The fusion of high spatial resolution panchromatic (PAN) data with simultaneously acquired multispectral (MS) data with the lower spatial resolution is a hot topic, which is often called pansharpening. In this article, we exploit the combination of machine learning techniques and fusion schemes introduced to address the pansharpening problem. In particular, deep convolutional neural networks (DCNNs) are proposed to solve this issue. The latter is combined first with the traditional component substitution and multiresolution analysis fusion schemes in order to estimate the nonlinear injection models that rule the combination of the upsampled low-resolution MS image with the extracted details exploiting the two philosophies. Furthermore, inspired by these two approaches, we also developed another DCNN for pansharpening. This is fed by the direct difference between the PAN image and the upsampled low-resolution MS image. Extensive experiments conducted both at reduced and full resolutions demonstrate that this latter convolutional neural network outperforms both the other detail injection-based proposals and several state-of-the-art pansharpening methods.

Image fusion in remote sensing by multi-objective deep learning

ABSTRACT This paper presents a multi-objective deep learning method for the purpose of achieving image fusion or pansharpening in remote sensing. This method uses a Denoising Autoencoder (DA). Two terms are added to the commonly used mean squared error loss function. The first term involves first applying a fractional-order superimposed gradient to the PANchromatic (PAN) image for extracting the high frequency information, and then by considering the difference between the network output and the edge map of the PANchromatic image. The second term involves the difference between the universal image quality index of the low resolution MultiSpectral (MS) image and the network output for each spectral band. These terms allow both the spatial and spectral information to be better preserved in the fused image. Experimental results on three public domain datasets for both low resolution and full resolution cases are reported based on commonly used objective metrics. Compared to the existing methods, the results obtained indicate the developed multi-objective method generates comparable results for the low resolution scenario and superior performance for the full resolution scenario.

Understanding the synergies of deep learning and data fusion of multispectral and panchromatic high resolution commercial satellite imagery for automated ice-wedge polygon detection

The utility of sheer volumes of very high spatial resolution (VHSR) commercial imagery in mapping the Arctic region is new and actively evolving. Commercial satellite sensors typically record image data in low-resolution multispectral (MS) and high-resolution panchromatic (PAN) mode. Spatial resolution is needed to accurately describe feature shapes and textural patterns, such as ice-wedge polygons (IWPs) that are rapidly transforming surface features due to degrading permafrost, while spectral resolution allows capturing of land-use and land-cover types. Data fusion, the process of combining PAN and MS images with complementary characteristics often serves as an integral component of remote sensing mapping workflows. The fusion process generates spectral and spatial artifacts that may affect the classification accuracies of subsequent automated image analysis algorithms, such as deep learning (DL) convolutional neural nets (CNN). We employed a detailed multidimensional assessment to understand the performances of an array of eight application-oriented data fusion algorithms when applied to VHSR image scenes for DLCNN-based mapping of ice-wedge polygons. Our findings revealed the scene dependency of data fusion algorithms and emphasized the need for careful selection of the proper algorithm. Results suggested that the fusion algorithms that preserve spatial character of original PAN imagery favor the DLCNN model performances. The choice of fusion approach needs to be considered of equal importance to the required training dataset for successful applications using DLCNN on VHRS imagery in order to enable an accurate mapping effort of permafrost thaw across the Arctic region.

Pansharpening Based on Joint-Guided Detail Extraction

Pansharpening is the process of fusing low spatial resolution multispectral (MS) images with high spatial resolution panchromatic (PAN) images, so as to obtain high spatial resolution multispectral (HRMS) images. In this article, a new pansharpening method based on a joint-guided detail extraction is proposed to maintain the spectral and spatial fidelity of a pansharpened image. First, to obtain details that are highly correlated with an MS image, a new PAN image is constructed and guided by the intensity component of the MS image through a variational model. The construction of the new PAN image improves the correlation between the PAN and MS images, and thus reduces the spectral distortion. The variational model is rapidly solved using the least-squares method. Second, to obtain accurate details from the new PAN image, the extraction of the details is guided by each band of the MS image through a regression model, which can further reduce the spatial distortion. The regression model is effectively solved using the gradient descend method. Finally, the details are injected into the upsampled MS image to obtain a fused image. Numerous experiments on the proposed approach were conducted and the results were compared with previous state-of-the-art pansharpening methods. The experimental results verify that the proposed method can efficiently achieve high-quality HRMS images.

Fusion of Hyperspectral and Multispectral Images Based on a Centralized Non-local Sparsity Model of Abundance Maps


 
 
 Objective: Hyperspectral (HS) imaging systems are commonly used in a diverse range of applications that involve detection and classification tasks. However, the low spatial resolution of hyperspectral images may limit the performance of the involved tasks in such applications. In the last years, fusing the information of an HS image with high spatial resolution multispectral (MS) or panchromatic (PAN) images has been widely studied to enhance the spatial resolution. Image fusion has been formulated as an inverse problem whose solution is an HS image which assumed to be sparse in an analytic or learned dictionary. This work proposes a non-local centralized sparse representation model on a set of learned dictionaries in order to regularize the conventional fusion problem.Methodology: The dictionaries are learned from the estimated abundance data taking advantage of the depth correlation between abundance maps and the non-local self- similarity over the spatial domain. Then, conditionally on these dictionaries, the fusion problem is solved by an alternating iterative numerical algorithm.Results: Experimental results with real data show that the proposed method outperforms the state-of-the-art methods under different quantitative assessments.Conclusions: In this work, we propose a hyperspectral and multispectral image fusion method based on a non-local centralized sparse representation on abundance maps. This model allows us to include the non-local redundancy of abundance maps in the fusion problem using spectral unmixing and improve the performance of the sparsity-based fusion approaches.
 
 

Remembering IRS-1C Mission: The “Game Changer” of Indian Earth Observation Program

India’s second-generation earth observation satellite series started with the launch of IRS-1C in December 1995, twenty-five years ago. It is considered to be a “game changer” for Indian earth observation program. With a 5.8 m panchromatic (PAN) camera onboard, IRS-1C became the highest spatial resolution civilian operational satellite at the time of launch and maintained this status till the launch of IKONOS satellite in 1999. IRS-1C also had a unique feature of 3-tier imaging geometry that drew the attention of global remote sensing user community. Other than the PAN camera, the 23.5 m multispectral camera including a shortwave infrared band and the high repetitive coarse-resolution two-band camera facilitated many newer applications and scientific investigations. Commercially, it was one of the most successful missions of the IRS program. The lessons learnt from this mission were even more significant, as these have led to many dedicated theme-missions subsequently.

Small Multispectral UAV Sensor and Its Image Fusion Capability in Cultural Heritage Applications

For many decades the multispectral images of the earth’s surface and its objects were taken from multispectral sensors placed on satellites. In recent years, the technological evolution produced similar sensors (much smaller in size and weight) which can be placed on Unmanned Aerial Vehicles (UAVs), thereby allowing the collection of higher spatial resolution multispectral images. In this paper, Parrot’s small Multispectral (MS) camera Sequoia+ is used, and its images are evaluated at two archaeological sites, on the Byzantine wall (ground application) of Thessaloniki city (Greece) and on a mosaic floor (aerial application) at the archaeological site of Dion (Greece). The camera receives RGB and MS images simultaneously, a fact which does not allow image fusion to be performed, as in the standard utilization procedure of Panchromatic (PAN) and MS image of satellite passive systems. In this direction, that is, utilizing the image fusion processes of satellite PAN and MS images, this paper demonstrates that with proper digital processing the images (RGB and MS) of small MS cameras can lead to a fused image with a high spatial resolution, which retains a large percentage of the spectral information of the original MS image. The high percentage of spectral fidelity of the fused images makes it possible to perform high-precision digital measurements in archaeological sites such as the accurate digital separation of the objects, area measurements and retrieval of information not so visible with common RGB sensors via the MS and RGB data of small MS sensors.

Hybrid modelling approach for water body change detection at Chalan Beel area in northern Bangladesh

Water is a strategic resource for both socio-economic development and human life. The current study has been carried out for spatio-temporal change detection of surface water bodies during winter period using hybrid modelling approach. The study area has fallen in the northern part of Bangladesh and is locally called Chalan Beel with 5 million of in habitants, a prominent intensive crop production, surface and groundwater irrigation, high evapotranspiration, and water scarcity. For the detection of water body changes, satellite images of 1999 and 2011 were used, and the following image fusion techniques were applied: (a) Gram-Schmidt (GS), (b) modified intensity hue saturation (IHS), (c) high-pass filter (HPF), and (d) wavelet. Landsat 7/ETM + panchromatic (PAN) band of 15 m × 15 m resolution in 1999 and Landsat 5/TM multispectral (MS) bands of 30 m × 30 m resolution in 2011 were allied each other to generate high-resolution image that contains information of two different years. The fused images were classified to extract the water bodies using four classification methods: (a) artificial neural network (ANN), (b) support vector machine (SVM) and (c) maximum likelihood (ML). To analyze the quality of the fused images, statistical calculation (quantitatively) and Laplacian edge detection (qualitatively) were used. To validate the fused image classification results, the multispectral images from 1999 and 2011 were again individually classified using principal component analysis (PCA), normalized difference water index (NDWI), and image differencing (ID) processes and compared with the previous classification. Surprisingly, the results showed that two-third of the areas dried up in 10 years.

The PAN and MS image fusion algorithm based on adaptive guided filtering and gradient information regulation

In recent years, with the improvement in the accuracy of remote sensing image classification and target recognition, the feature level fusion technology of remote sensing images has attracted much attention and become a research hotspot. However, this kind of fusion technology is not as mature as pixel-level fusion technology, and there are still many problems to be solved. This paper proposes a multi-spectral (MS) and panchromatic (PAN) image fusion algorithm based on adaptive textural feature extraction and information injection regulation. The fusion algorithm includes two stages. The first stage extracts the textural details of high-resolution PAN images. In this stage, based on the sensitivity of the remote sensing images to the gray-level co-occurrence matrix (GLCM), an adaptive guided filter (AGIF) scheme for remote sensing images based on the GLCM is proposed. The feature information of the textures and details of the PAN image was fully extracted. The second stage injects the extracted feature information of the PAN image into an MS image. In this stage, a decision map based on the MS image gradient domain and a weighted matrix based on the gradient entropy measure were proposed in order to, respectively, realize the adaptability of the feature injection location selection and regulate the intensity of the injected information to the MS image. This ensures the rationality of the injection of the textural information and avoids noise, patches and other information interference. The proposed algorithm has the advantages of fully extracting the textural features of high-resolution PAN images, adaptively adjusting the injection position and intensity when injecting the feature information into an MS image, and providing the fused image with clear features. On the premise of effectively maintaining the spectral information quality, the spatial resolution of the fused image is improved. A large number of simulation experiments verify the effectiveness of the proposed method.

Pansharpening: Context-Based Generalized Laplacian Pyramids by Robust Regression

Pansharpening refers to the combination of panchromatic (PAN) and multispectral (MS) images, designed to obtain a fused product retaining the fine spatial resolution of the former and the high spectral content of the latter. One of the most popular and successful approaches to pansharpening is the method known as context-based generalized Laplacian pyramid, which requires as a key ingredient for the estimation of the so-called injection coefficients. In this article, we propose the adoption of robust techniques for the estimation of the injection coefficients and detection strategies to select the clusters for which robust regression is needed, providing a suitable balancing between fusion performance and computational burden. Experimental results conducted on five real data sets acquired by the sensors QuickBird, WorldView-3, and WorldView-4, show the superiority of the proposed method with respect to current state-of-the-art pansharpening techniques.

PWNet: An Adaptive Weight Network for the Fusion of Panchromatic and Multispectral Images

Pansharpening is a typical image fusion problem, which aims to produce a high resolution multispectral (HRMS) image by integrating a high spatial resolution panchromatic (PAN) image with a low spatial resolution multispectral (MS) image. Prior arts have used either component substitution (CS)-based methods or multiresolution analysis (MRA)-based methods for this propose. Although they are simple and easy to implement, they usually suffer from spatial or spectral distortions and could not fully exploit the spatial and/or spectral information existed in PAN and MS images. By considering their complementary performances and with the goal of combining their advantages, we propose a pansharpening weight network (PWNet) to adaptively average the fusion results obtained by different methods. The proposed PWNet works by learning adaptive weight maps for different CS-based and MRA-based methods through an end-to-end trainable neural network (NN). As a result, the proposed PWN inherits the data adaptability or flexibility of NN, while maintaining the advantages of traditional methods. Extensive experiments on data sets acquired by three different kinds of satellites demonstrate the superiority of the proposed PWNet and its competitiveness with the state-of-the-art methods.

Image pan-sharpening using enhancement based approaches in remote sensing

This paper proposes to do image enhancement before pan-sharpening; that is, the image enhancement techniques are used as a pre-processing step. The image enhancement techniques are proposed in two domains, same-domain and cross-domain. In the same-domain methods, the image enhancement techniques (such as Laplacian, Unsharp) are simply applied to multispectral (MS) and panchromatic (PAN) images to sharpen both images in the spatial domain. While in cross-domain, a novel hybrid combination of Laplacian Filter (LF) and Discrete Fourier Transformation (DFT) image sharpening technique is introduced. After image enhancement, the powerful Matting Model (MM) pan-sharpening technique is used to fuse both the enhanced images and produce a resultant image with the high spatial and spectral resolutions. The experimental results of the proposed approach outperform the others as compared to the state-of-art techniques over three datasets. The results are evaluated, considering both Qualitative and Quantitative evaluation metrics.