Pan-sharpened Image Research Articles

A novel cross fusion model with fine-grained detail reconstruction for remote sensing image pan-sharpening

ABSTRACT Pan-sharpening aims to obtain high resolution multispectral (HRMS) images by integrating the information in the panchromatic and multispectral images. Existing pan-sharpening methods have demonstrated impressive sharpening performance. However, these methods inherently overlook the complementary characteristics and interaction between diverse source images, resulting in sharpened outcomes accompanied by distortion. To solve the above problems, we construct a novel cross fusion model with fine-grained detail reconstruction from the perspective of frequency-domain. The motivation of the model is twofold: (1) to reconstruct spatial detail representations from diverse source images, laying the foundation for the generation of fine details in the subsequent fused images; and (2) to enhance the interaction between diverse source features during the fusion process in order to attain high-fidelity fusion outcomes. Based on the theoretical model, we develop a frequency-spectral dual domain cross fusion network (CF2N) utilizing the deep learning technique. Consequently, the CF2N consist of two main stages, namely frequency-domain dominated detail reconstruction (FD2R) and frequency-spectral cross fusion (FSCF). Specifically, a more reasonable reconstruction of fine frequency details in HRMS can be achieved by performing adaptive weighted fusion of frequency details in the FD2R stage. Furthermore, the FSCF module, which seamlessly integrates frequency- and spectral-domain details in a highly interactive cross fusion manner. As a result, the CF2N possesses the capability to attain high frequency-spectral fidelity results with excellent interpretability. Extensive experiments show the superior performance of ours over state of the art, while maintaining high efficiency. All implementations of this work will be published at our website.

Self Organizing Map based Land Cover Clustering for Decision-Level Jaccard Index and Block Activity based Pan-Sharpened Images

Self Organizing Map based Land Cover Clustering for Decision-Level Jaccard Index and Block Activity based Pan-Sharpened Images

A combined Remote Sensing and GIS-based method for Local Climate Zone mapping using PRISMA and Sentinel-2 imagery

In the last decade, several methods have been developed for Local Climate Zone (LCZ) mapping, encompassing Remote Sensing and Geographic Information Systems (GIS) −based procedures. Combined approaches have also been proposed to compensate for intrinsic limitations that characterized their separate application. Recent work has disclosed the potential of hyperspectral satellite imagery for improving LCZ identification. However, the use of hyperspectral data for LCZ mapping is yet to be fully unfolded. A combined Remote Sensing and GIS-based method for LCZ mapping is proposed to exploit the integration of hyperspectral PRISMA and multispectral Sentinel-2 images with ancillary urban canopy parameter layers. Random Forest algorithm is applied to the feature sets to obtain the LCZ classification. The method is tested on the Metropolitan City of Milan (Italy), for the period from February to August 2023. A spectral separability analysis is carried out to investigate the improvement in LCZ identification using PRISMA in comparison to Sentinel-2 data, as well as improvements in LCZ spectral separability on PRISMA pan-sharpened images. The resulting maps’ quality is evaluated by extracting accuracy metrics and performing inter-comparisons with maps computed from the LCZ Generator benchmark tool. Inter-comparisons yield promising results with a mean Overall Accuracy increase of 16% using PRISMA for each LCZ class. Furthermore, we find that PRISMA improves the detection of LCZs compared to Sentinel-2, with a mean Overall Accuracy increase of 5%, in line with the higher spectral separability of PRISMA spectral signatures computed on the training samples.

PRISMA Hyperspectral Satellite Imagery Application to Local Climate Zones Mapping

Abstract. The urban heat island effect exacerbates the vulnerability of cities to climate change, emphasizing the need for sustainable urban planning driven by data evidence. In the last decade, the Local Climate Zone (LCZ) model emerged as a key tool for categorizing urban landscapes, aiding in the development of urban temperature mitigation strategies. In this work, the contribution of hyperspectral satellite imagery to LCZ mapping, leveraging the Italian Space Agency (ASI)’s PRISMA satellite, is investigated. Mapping performances are compared with traditional multispectral-based LCZ mapping using Sentinel-2 satellite imagery. The Random Forest algorithm is utilized for LCZ classification, with evaluation conducted through spectral separability analysis and accuracy assessment between PRISMA and Sentinel-2 derived LCZ maps as well as with the benchmark LCZ Generator mapping tool. An initial experiment on the effect of PRISMA image pan-sharpening on LCZ spectral separability is also presented. Results obtained for Milan (Northern Italy) demonstrate the potential of hyperspectral imagery in enhancing LCZ identification compared to multispectral data, with promising improvements in LCZ maps overall accuracy. Finally, air temperature patterns within each LCZ class are explored, qualitatively confirming the influence of urban morphology on thermal comfort.

Unsupervised Pan-Sharpening via Mutually Guided Detail Restoration

Pan-sharpening is a task that aims to super-resolve the low-resolution multispectral (LRMS) image with the guidance of a corresponding high-resolution panchromatic (PAN) image. The key challenge in pan-sharpening is to accurately modeling the relationship between the MS and PAN images. While supervised deep learning methods are commonly employed to address this task, the unavailability of ground-truth severely limits their effectiveness. In this paper, we propose a mutually guided detail restoration method for unsupervised pan-sharpening. Specifically, we treat pan-sharpening as a blind image deblurring task, in which the blur kernel can be estimated by a CNN. Constrained by the blur kernel, the pan-sharpened image retains spectral information consistent with the LRMS image. Once the pan-sharpened image is obtained, the PAN image is blurred using a pre-defined blur operator. The pan-sharpened image, in turn, is used to guide the detail restoration of the blurred PAN image. By leveraging the mutual guidance between MS and PAN images, the pan-sharpening network can implicitly learn the spatial relationship between the two modalities. Extensive experiments show that the proposed method significantly outperforms existing unsupervised pan-sharpening methods.

A High-Resolution Hyperspectral Imaging System for the Retina.

In this study, we developed an imaging system that can acquire and produce high-resolution hyperspectral images of the retina. Our system combines the view from a high-resolution RGB camera and a snapshot hyperspectral camera together. The method is fast and can be constructed into a compact imaging device. We tested our system by imaging a calibrated color chart, biological tissues ex vivo, and a phantom of the human retina. By using image pansharpening methods, we were able to produce a high-resolution hyperspectral image. The images from the hyperspectral camera alone have a spatial resolution of 0.2 mm/pixel, whereas the pansharpened images have a spatial resolution of 0.1 mm/pixel, a 2x increase in spatial resolution. Our method has the potential to capture images of the retina rapidly. Our method preserves both the spatial and spectral fidelity, as shown by comparing the original hyperspectral images with the pansharpened images. The high-resolution hyperspectral imaging device can have a variety of applications in retina examinations.

SWPanGAN: A hybrid generative adversarial network for pansharpening

AbstractPansharpening is a vital technique in remote sensing that combines a low‐resolution multi‐spectral image with its corresponding panchromatic image to obtain a high‐resolution multi‐spectral image. Despite its potential benefits, the challenge lies in extracting features from the source images and eliminating artefacts in the fused images. In response to the challenge, a hybrid generative adversarial network‐based model, termed SWPanGAN, is proposed. For better feature extraction, the conventional convolution neural network is replaced with a Swin transformer in the generator, which provides the generator with the ability to model long‐range dependencies. Additionally, to suppress artefacts, a wavelet‐based discriminator is proposed for effectively distinguishing the frequency discrepancy. With these modifications, both the generator and discriminator networks of SWPanGAN are enhanced. Extensive experiments illustrate that our SWPanGAN can generate high‐quality pansharpening images and surpass other state‐of‐the‐art methods.

GSA-SiamNet: A Siamese Network with Gradient-Based Spatial Attention for Pan-Sharpening of Multi-Spectral Images

Pan-sharpening is a fusion process that combines a low-spatial resolution, multi-spectral image that has rich spectral characteristics with a high-spatial resolution panchromatic (PAN) image that lacks spectral characteristics. Most previous learning-based approaches rely on the scale-shift assumption, which may not be applicable in the full-resolution domain. To solve this issue, we regard pan-sharpening as a multi-task problem and propose a Siamese network with Gradient-based Spatial Attention (GSA-SiamNet). GSA-SiamNet consists of four modules: a two-stream feature extraction module, a feature fusion module, a gradient-based spatial attention (GSA) module, and a progressive up-sampling module. In the GSA module, we use Laplacian and Sobel operators to extract gradient information from PAN images. Spatial attention factors, learned from the gradient prior, are multiplied during the feature fusion, up-sampling, and reconstruction stages. These factors help to keep high-frequency information on the feature map as well as suppress redundant information. We also design a multi-resolution loss function that guides the training process under the constraints of both reduced- and full-resolution domains. The experimental results on WorldView-3 satellite images obtained in Moscow and San Juan demonstrate that our proposed GSA-SiamNet is superior to traditional and other deep learning-based methods.

Pan-sharpening via intrinsic decomposition knowledge distillation

Existing deep-learning-based pan-sharpening strategies mainly involve the fusion of panchromatic and multispectral (MS) information at both the pixel and feature levels. In this paper, we hypothesize that the MS image can be expressed as the multiplication of reflectance and illumination components, and that the reflection components of low-resolution (LR) MS and high-resolution (HR) MS images are invariant. Here, the spectral reflection component can effectively describe the spectral response of an object, while the illumination component can effectively describe its texture. Based on this hypothesis, we propose a novel and concise pan-sharpening framework called intrinsic decomposition knowledge distillation. Specifically, the teacher network decomposes the HR MS image into reflectance and illumination components, which are then combined in the student network with the reflectance component and the enhanced illumination component from LR MS to reconstruct the pan-sharpened image. To approximate the component distributions from the teacher network, we introduce a novel three-stage knowledge distillation strategy that can transfer knowledge about the relationships between components and constrain the student network. Our quantitative and qualitative comparisons demonstrate the reasonableness of our hypothesis and the effectiveness of our proposed method in significantly improving perception quality.

Deep Spectral Blending Network for Color Bleeding Reduction in PAN-Sharpening Images

Deep Spectral Blending Network for Color Bleeding Reduction in PAN-Sharpening Images

Two-discriminators-deep residual GAN hyperspectral image pan-sharpening

Two-discriminators-deep residual GAN hyperspectral image pan-sharpening

MMAPP: Multi-branch and Multi-scale Adaptive Progressive Pyramid Network for Multispectral Image Pansharpening

MMAPP: Multi-branch and Multi-scale Adaptive Progressive Pyramid Network for Multispectral Image Pansharpening

Remote Sensing Image Pansharpening using Deep Internal Learning with Residual Double-Attention Network

Remote Sensing Image Pansharpening using Deep Internal Learning with Residual Double-Attention Network

Pan-Sharpening Network of Multi-Spectral Remote Sensing Images Using Two-Stream Attention Feature Extractor and Multi-Detail Injection (TAMINet)

Achieving a balance between spectral resolution and spatial resolution in multi-spectral remote sensing images is challenging due to physical constraints. Consequently, pan-sharpening technology was developed to address this challenge. While significant progress was recently achieved in deep-learning-based pan-sharpening techniques, most existing deep learning approaches face two primary limitations: (1) convolutional neural networks (CNNs) struggle with long-range dependency issues, and (2) significant detail loss during deep network training. Moreover, despite these methods’ pan-sharpening capabilities, their generalization to full-sized raw images remains problematic due to scaling disparities, rendering them less practical. To tackle these issues, we introduce in this study a multi-spectral remote sensing image fusion network, termed TAMINet, which leverages a two-stream coordinate attention mechanism and multi-detail injection. Initially, a two-stream feature extractor augmented with the coordinate attention (CA) block is employed to derive modal-specific features from low-resolution multi-spectral (LRMS) images and panchromatic (PAN) images. This is followed by feature-domain fusion and pan-sharpening image reconstruction. Crucially, a multi-detail injection approach is incorporated during fusion and reconstruction, ensuring the reintroduction of details lost earlier in the process, which minimizes high-frequency detail loss. Finally, a novel hybrid loss function is proposed that incorporates spatial loss, spectral loss, and an additional loss component to enhance performance. The proposed methodology’s effectiveness was validated through experiments on WorldView-2 satellite images, IKONOS, and QuickBird, benchmarked against current state-of-the-art techniques. Experimental findings reveal that TAMINet significantly elevates the pan-sharpening performance for large-scale images, underscoring its potential to enhance multi-spectral remote sensing image quality.

Pansharpening of remote sensing images using dominant pixels

Pansharpening refers to the synthesis of super-resolution multispectral images (i.e., pansharpened images, PIs) designed to overcome the technical constraints of Earth Observation Satellites. A PI is synthesized by fusing the high-resolution chromatic information carried by the multispectral image with the high-resolution spatial information carried by the panchromatic image. This process generates precise and comprehensive data suitable for diverse applications, such as land cover mapping, urban planning, and natural resource management. Various histogram transformation techniques are utilized in pansharpening methods to improve the chromatic fidelity of PIs. However, many existing histogram transformation techniques are susceptible to chromatic distortions. In this paper, a novel pansharpening method named “Pansharpening Using Dominant Pixels” (PDP) is introduced. PDP employs a new method for histogram transformation that preserves chromatic information. Furthermore, PDP is structurally simple, easy to implement, fast, and capable of producing high-quality pansharpened images. Six Remote Sensing images and seventeen pansharpening methods were used in experiments to examine PDP’s success in generating pansharpened images. The experimental results demonstrate that PDP statistically outperforms the comparison methods, synthesizing PIs with high spectral and spatial fidelity.

A variational driven optimization framework for pansharpening of multispectral images

ABSTRACT Pansharpening is an established remote sensing image fusion technique that yields a high-resolution multispectral (HRMS) image. Despite the fact that the advanced technologies like sparse coding and deep learning have achieved a remarkable improvement in solving the pansharpening problem, a unified model is required to further enhance the fusion quality. The variational optimisation (VO) mechanism has gained interest of most of the researchers in recent years. In this article, pansharpening is designated as a constrained optimisation problem with a data generative term and two regularisers to ameliorate spatial details and spectral information. The gradient information is exploited to impart the spatial details from the panchromatic image to the fused image. The correlation among multispectral image bands inspired to promote the spectral quality of the HRMS image as well as to reduce the distortion. Consequently, the optimisation problem is efficiently solved for the required HRMS image using the operator splitting approach. The extensive experiments performed in accordance with the well-known protocols rationalise that the proposed model outperforms most of the state-of-the-art methods in terms of objective metrics and visual outcomes.

Spectral Profile Partial Least-Squares (SP-PLS): Local multivariate pansharpening on spectral profiles

The compatibility of multispectral (MS) pansharpening algorithms with hyperspectral (HS) data is limited. With the recent development in HS satellites, there is a need for methods that can provide high spatial and spectral fidelity in both HS and MS scenarios.The present article presents a fast pansharpening method, based on the division of similar hyperspectral data in spectral subgroups using k-means clustering and Spectral Angle Mapper (SAM) profiling. Local Partial Least-Square (PLS) models are calibrated for each spectral subgroup against the respective pixels of the panchromatic image. The models are inverted to retrieve high-resolution pansharpened images. The method is tested against different methods that are able to handle both MS and HS pansharpening and assessed using reduced- and full-resolution evaluation methodologies. Based on a statistical multivariate approach, the proposed method is able to render uncertainty maps for spectral or spatial fidelity - functionality not reported in any other pansharpening study.

Salt Land classification based on pansharpening images using Random Forest algorithm

ABSTRACT The analysis and policymaking of national salt production require quality Salt Land maps with more detailed object information. This study successfully classified Salt Land into Active and Inactive based on pansharpening images using random forest algorithm. The results show: 1) the best pansharpening method is UNB PanSharp; 2) the texture variables have more significant contributions to enhance classification accuracy; 3) scheme 5 is the most efficient and recommended scheme in research, achieves an OA of 66.699% and a Kappa coefficient of 39.4%. This study can provide a good reference for developing Salt Land mapping and policies for practitioners or stakeholders.

A Swin Transformer with Dynamic High-Pass Preservation for Remote Sensing Image Pansharpening

Pansharpening is a technique used in remote sensing to combine high-resolution panchromatic (PAN) images with lower resolution multispectral (MS) images to generate high-resolution multispectral images while preserving spectral characteristics. Recently, convolutional neural networks (CNNs) have been the mainstream in pansharpening by extracting the deep features of PAN and MS images and fusing these abstract features to reconstruct high-resolution details. However, they are limited by the short-range contextual dependencies of convolution operations. Although transformer models can alleviate this problem, they still suffer from weak capability in reconstructing high-resolution detailed information from global representations. To this end, a novel Swin-transformer-based pansharpening model named SwinPAN is proposed. Specifically, a detail reconstruction network (DRNet) is developed in an image difference and residual learning framework to reconstruct the high-resolution detailed information from the original images. DRNet is developed based on the Swin Transformer with a dynamic high-pass preservation module with adaptive convolution kernels. The experimental results on three remote sensing datasets with different sensors demonstrate that the proposed approach performs better than state-of-the-art networks through qualitative and quantitative analysis. Specifically, the generated pansharpening results contain finer spatial details and richer spectral information than other methods.

Deep Learning-Based Land Classification: A CNN Architecture for High-Accuracy Land Use Mapping

This research paper focuses on the important method of image matching for collecting ground control points and automated precise geo-registration of high-resolution satellite imagery. The study aims to improve the matching success rate between incoming satellite images and reference chips generated from aerial color ortho-images by using pan-sharpened satellite images. The results show that the use of pan-sharpened images leads to higher matching success rates due to similar spectral range and higher spatial resolution. The paper also highlights the significance of accurate land cover information for various geospatial applications such as agriculture, environmental and urban management. Traditional methods of gathering land cover information are time-consuming and involve physical labor, making automated methods a more efficient and practical solution.