Hyperspectral Pansharpening Research Articles

Deep Learning Hyperspectral Pansharpening on Large-Scale PRISMA Dataset

Hyperspectral pansharpening is crucial for the improvement of the usability of images in various applications. However, it remains underexplored due to a scarcity of data. The primary goal of pansharpening is to enhance the spatial resolution of hyperspectral images by reconstructing missing spectral information without compromising consistency with the original data. This paper addresses the data gap by presenting a new hyperspectral dataset specifically designed for pansharpening and the evaluation of several deep learning strategies using this dataset. The new dataset has two crucial features that make it invaluable for deep learning hyperspectral pansharpening research. (1) It presents the highest cardinality of images in the state of the art, making it the first statistically relevant dataset for hyperspectral pansharpening evaluation, and (2) it includes a wide variety of scenes, ensuring robust generalization capabilities for various approaches. The data, collected by the ASI PRISMA satellite, cover about 262,200 km2 and their heterogeneity is ensured by a random sampling of the Earth’s surface. The analysis of the deep learning methods consists in the adaptation of these approaches to the PRISMA hyperspectral data and the quantitative and qualitative evaluation of their performance in this new scenario. The investigation included two settings: Reduced Resolution (RR) to evaluate the techniques in a controlled environment and Full Resolution (FR) for a real-world evaluation. In addition, for the sake of completeness, we have also included machine-learning-free approaches in both scenarios. Our comprehensive analysis reveals that data-driven neural network methods significantly outperform traditional approaches, demonstrating a superior adaptability and performance in hyperspectral pansharpening under both RR and FR protocols.

Multi-task multi-objective evolutionary network for hyperspectral image classification and pansharpening

Multi-task learning has commonly been used and performed well at joint visual perception tasks. Hyperspectral pansharpening (HP) and hyperspectral classification (HC) tasks extract high-frequency information to enhance edges and classify samples, offering potential for performance improvements in multi-task learning. However, differences between tasks can make it challenging to balance their performances. To address this challenge, this paper proposes a multi-task multi-objective evolutionary network (DMOEAD) for joint learning of HC and HP. A multi-task sufficiency-and-diversity sampling method is designed to unify the heterogeneity of sample construction between two types of tasks. Two types of task-specific networks are constructed to decompose high-frequency information. Further, a collaborative learning module is designed to dynamically learn complementary high-frequency information from another task in different layers. To be compatible with the optimization direction of two types of tasks, multi-task optimization is realized using a deep multi-objective evolutionary algorithm (DMEO). In the DMEO, the set of parameters of the DMOEAD is regarded as an individual. A deep mutation operator is designed and used for network optimization, which accelerates large-scale network parameter searching. The DMEO can coordinate the differences between multiple tasks and provide a set of Pareto network parameter solutions. Finally, the experimental results demonstrate that the proposed method can significantly enhance the performance of both pansharpening and classification tasks.

AIDB-Net: An Attention-Interactive Dual-Branch Convolutional Neural Network for Hyperspectral Pansharpening

Despite notable advancements achieved on Hyperspectral (HS) pansharpening tasks through deep learning techniques, previous methods are inherently constrained by convolution or self-attention intrinsic defects, leading to limited performance. In this paper, we proposed an Attention-Interactive Dual-Branch Convolutional Neural Network (AIDB-Net) for HS pansharpening. Our model purely consists of convolutional layers and simultaneously inherits the strengths of both convolution and self-attention, especially the modeling of short- and long-range dependencies. Specially, we first extract, tokenize, and align the hyperspectral image (HSI) and panchromatic image (PAN) by Overlapping Patch Embedding Blocks. Then, we specialize a novel Spectral-Spatial Interactive Attention which is able to globally interact and fuse the cross-modality features. The resultant token-global similarity scores can guide the refinement and renewal of the textural details and spectral characteristics within HSI features. By deeply combined these two paradigms, our AIDB-Net significantly improve the pansharpening performance. Moreover, with the acceleration by the convolution inductive bias, our interactive attention can be trained without large scale dataset and achieves competitive time cost with its counterparts. Compared with the state-of-the-art methods, our AIDB-Net makes 5.2%, 3.1%, and 2.2% improvement on PSNR metric on three public datasets, respectively. Comprehensive experiments quantitatively and qualitatively demonstrate the effectiveness and superiority of our AIDB-Net.

A hyperspectral pansharpening method using retrain transformer network for remote sensing images in UAV communications system

A hyperspectral pansharpening method using retrain transformer network for remote sensing images in UAV communications system

Feature Pyramid Fusion Network for Hyperspectral Pansharpening.

Hyperspectral (HS) pansharpening aims at fusing an observed HS image with a panchromatic (PAN) image, to produce an image with the high spectral resolution of the former and the high spatial resolution of the latter. Most of the existing convolutional neural networks (CNNs)-based pansharpening methods reconstruct the desired high-resolution image from the encoded low-resolution (LR) representation. However, the encoded LR representation captures semantic information of the image and is inadequate in reconstructing fine details. How to effectively extract high-resolution and LR representations for high-resolution image reconstruction is the main objective of this article. In this article, we propose a feature pyramid fusion network (FPFNet) for pansharpening, which permits the network to extract multiresolution representations from PAN and HS images in two branches. The PAN branch starts from the high-resolution stream that maintains the spatial resolution of the PAN image and gradually adds LR streams in parallel. The structure of the HS branch remains highly consistent with that of the PAN branch, but starts with the LR stream and gradually adds high-resolution streams. The representations with corresponding resolutions of PAN and HS branches are fused and gradually upsampled in a coarse to fine manner to reconstruct the high-resolution HS image. Experimental results on three datasets demonstrate the significant superiority of the proposed FPFNet over the state-of-the-art methods in terms of both qualitative and quantitative comparisons.

Tree-Structured Neural Network for Hyperspectral Pansharpening

Tree-Structured Neural Network for Hyperspectral Pansharpening

Two Spectral–Spatial Implicit Neural Representations for Arbitrary-Resolution Hyperspectral Pansharpening

Two Spectral–Spatial Implicit Neural Representations for Arbitrary-Resolution Hyperspectral Pansharpening

Dual-branch U-Net with hybrid attention for hyperspectral pansharpening

Dual-branch U-Net with hybrid attention for hyperspectral pansharpening

Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer

Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer

An Interpretable Unsupervised Unrolling Network for Hyperspectral Pansharpening.

Existing deep convolutional neural networks (CNNs) have recently achieved great success in pansharpening. However, most deep CNN-based pansharpening models are based on "black-box" architecture and require supervision, making these methods rely heavily on the ground-truth data and lose their interpretability for specific problems during network training. This study proposes a novel interpretable unsupervised end-to-end pansharpening network, called as IU2PNet, which explicitly encodes the well-studied pansharpening observation model into an unsupervised unrolling iterative adversarial network. Specifically, we first design a pansharpening model, whose iterative process can be computed by the half-quadratic splitting algorithm. Then, the iterative steps are unfolded into a deep interpretable iterative generative dual adversarial network (iGDANet). Generator in iGDANet is interwoven by multiple deep feature pyramid denoising modules and deep interpretable convolutional reconstruction modules. In each iteration, the generator establishes an adversarial game with the spatial and spectral discriminators to update both spectral and spatial information without ground-truth images. Extensive experiments show that, compared with the state-of-the-art methods, our proposed IU2PNet exhibits very competitive performance in terms of quantitative evaluation metrics and qualitative visual effects.

An Optimality Property of Matrix-R Theorem, its Extension, and the Application to Hyperspectral Pan-sharpening

An Optimality Property of Matrix-R Theorem, its Extension, and the Application to Hyperspectral Pan-sharpening

Spectral Profile Partial Least-Squares (SP-PLS): Local multivariate pansharpening on spectral profiles

The compatibility of multispectral (MS) pansharpening algorithms with hyperspectral (HS) data is limited. With the recent development in HS satellites, there is a need for methods that can provide high spatial and spectral fidelity in both HS and MS scenarios.The present article presents a fast pansharpening method, based on the division of similar hyperspectral data in spectral subgroups using k-means clustering and Spectral Angle Mapper (SAM) profiling. Local Partial Least-Square (PLS) models are calibrated for each spectral subgroup against the respective pixels of the panchromatic image. The models are inverted to retrieve high-resolution pansharpened images. The method is tested against different methods that are able to handle both MS and HS pansharpening and assessed using reduced- and full-resolution evaluation methodologies. Based on a statistical multivariate approach, the proposed method is able to render uncertainty maps for spectral or spatial fidelity - functionality not reported in any other pansharpening study.

HyperRefiner: a refined hyperspectral pansharpening network based on the autoencoder and self-attention

ABSTRACT Deep learning has been developed to generate promising super resolution hyperspectral imagery by fusing hyperspectral imagery with the panchromatic band. However, it is still challenging to maintain edge spectral information in the necessary upsampling processes of these approaches, and difficult to guarantee effective feature extraction. This study proposes a pansharpening network denoted as HyperRefiner that consists of, (1) a well performing upsampling network SRNet, in which the dual attention block and refined attention block are cascaded to accomplish the extraction and fusion of features; (2) a spectral autoencoder that is embedded to perform dimensionality reduction under constrained feature extraction; and (3) the optimization module which performs self-attention at the pixel and feature levels. A comparison with several state-of-the-art models reveals that HyperRefiner can improve the quality of the fused image. Specifically, compared to the single-head HyperTransformer and with the Chikusei dataset, our network improved the Peak Signal-to-Noise Ratio, Erreur Relative Globale Adimensionnelle de Synthèse and Spectral Angle Mapper by 0.86%, 3.62%, and 2.09%, and reduce the total memory, floating point operations, model parameters and computation time by 41%, 75%, 86% and 46%, respectively. The experimental results show that HyperRefiner outperforms several networks and demonstrates its usefulness in hyperspectral image fusion. The code is publicly available at https://github.com/zsspo/Fusion_HyperRefiner.

Dynamic Hyperspectral Pansharpening CNNs

Hyperspectral (HS) pansharpening seeks to integrate low spatial resolution HS (LRHS) images with connected panchromatic (PAN) images to produce high spatial resolution HS (HRHS) images. Traditional pansharpening convolutional neural networks (CNNs) directly map LRHS and PAN images into HRHS images under fixed network parameters, which imply static pansharpening rules. However, real-world HS data are often characterized by spatial variations and intuitively the pansharpening rules should be dynamic. To deal with the dilemma, in this paper we develop dynamic HS pansharpening CNNs. We first specify the concepts of dynamic pansharpening and static pansharpening. Then, we propose a learn-to-learn oriented pansharpening CNN paradigm, which aims to learn a how-to-learn rule to produce spatially adaptive pansharpening rules and comprises three stages of preliminary fusion, scene-sensitive modulation and spectral reconstruction. Finally, following the paradigm, we design two groups of dynamic pansharpening CNNs (DyPNNs), i.e., internal-connection-based and external-connection-based. They involve various spatial modulations, including spatial affine transform (AT), spatial dynamic convolution (DC) or improved spatial attention (SA), and thus consist of six specific DyPNNs: IC-AT-DyPNN, IC-DC-DyPNN, IC-SA-DyPNN, EC-AT-DyPNN, EC-DC-DyPNN and EC-SA-DyPNN. Experimental results on several HS datasets verify the effectiveness of the proposed DyPNNs in terms of both the spatial reconstruction and spectral fidelity.

Panchromatic and Hyperspectral Image Fusion: Outcome of the 2022 WHISPERS Hyperspectral Pansharpening Challenge

This article presents the scientific outcomes of the 2022 Hyperspectral Pansharpening Challenge organized by the 12th IEEE Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (IEEE WHISPERS 2022). The 2022 Hyperspectral Pansharpening Challenge aims at fusing a panchromatic image with hyperspectral data to get a high spatial resolution hyperspectral cube with the same spatial resolution of the panchromatic image while preserving the spectral information of hyperspectral data. Four datasets acquired by the PRISMA mission owned and managed by the Italian Space Agency have been prepared for participants. They are made available for the benefit of the scientific community. Each dataset contains a panchromatic image and a hyperspectral cube with different spatial resolutions. More than 100 registrations have been received for the event. Four teams submitted their outcomes. Since no team actually outperformed the baseline provided by the organizers, the challenge was declared inconclusive and no winner was recognized.

Generative Dual-Adversarial Network With Spectral Fidelity and Spatial Enhancement for Hyperspectral Pansharpening.

Hyperspectral (HS) pansharpening is of great importance in improving the spatial resolution of HS images for remote sensing tasks. HS image comprises abundant spectral contents, whereas panchromatic (PAN) image provides spatial information. HS pansharpening constitutes the possibility for providing the pansharpened image with both high spatial and spectral resolution. This article develops a specific pansharpening framework based on a generative dual-adversarial network (called PS-GDANet). Specifically, the pansharpening problem is formulated as a dual task that can be solved by a generative adversarial network (GAN) with two discriminators. The spatial discriminator forces the intensity component of the pansharpened image to be as consistent as possible with the PAN image, and the spectral discriminator helps to preserve spectral information of the original HS image. Instead of designing a deep network, PS-GDANet extends GANs to two discriminators and provides a high-resolution pansharpened image in a fraction of iterations. The experimental results demonstrate that PS-GDANet outperforms several widely accepted state-of-the-art pansharpening methods in terms of qualitative and quantitative assessment.

SSML: Spectral-Spatial Mutual-Learning-Based Framework for Hyperspectral Pansharpening

This paper considers problems associated with the large size of the hyperspectral pansharpening network and difficulties associated with learning its spatial-spectral features. We propose a deep mutual-learning-based framework (SSML) for spectral-spatial information mining and hyperspectral pansharpening. In this framework, a deep mutual-learning mechanism is introduced to learn spatial and spectral features from each other through information transmission, which achieves better fusion results without entering too many parameters. The proposed SSML framework consists of two separate networks for learning spectral and spatial features of HSIs and panchromatic images (PANs). A hybrid loss function containing constrained spectral and spatial information is designed to enforce mutual learning between the two networks. In addition, a mutual-learning strategy is used to balance the spectral and spatial feature learning to improve the performance of the SSML path compared to the original. Extensive experimental results demonstrated the effectiveness of the mutual-learning mechanism and the proposed hybrid loss function for hyperspectral pan-sharpening. Furthermore, a typical deep-learning method was used to confirm the proposed framework’s capacity for generalization. Ideal performance was observed in all cases. Moreover, multiple experiments analysing the parameters used showed that the proposed method achieved better fusion results without adding too many parameters. Thus, the proposed SSML represents a promising framework for hyperspectral pansharpening.

CMNet: Classification-oriented multi-task network for hyperspectral pansharpening

Hyperspectral pansharpening (HP) can fuse different kinds of remote sensing images for the downstream classification or detection task. Deep learning methods have made rapid development in HP. However, most deep learning-based HP methods design the network architecture without considering the specific downstream task. When the pansharpening image is transferred to the downstream tasks, it may lead to poor performance on the downstream tasks. To solve this problem, a novel classification-oriented multi-task network (CMNet) is proposed for HP. CMNet combines the classification and HP networks in a multi-task learning way. In the classification network, class-specific and spatial-difference contrastive constraints are defined to enforce the extraction module to extract discriminative features for classification. In the HP network, a fusion module with an identical structure to the extraction module is constructed to extract fused features for reconstruction. Then, multi-gradient learning is designed to optimize the HP network under the guidance of the classification network. It adopts gradient mapping to consider the conflicting or collaborative situations between these different tasks. In this way, the fusion module in the HP network learns discriminative feature extraction ability from the classification network, which makes the HP network realize classification-oriented HP. Experimental results demonstrate that the proposed method markedly improves the pansharpening performance and the pansharpening is also able to elevate the classification performance.

Multi-level features fusion via cross-layer guided attention for hyperspectral pansharpening

Recently, the successful applications of convolutional neural network (CNN) in computer vision have attracted considerable attentions. Particularly, deep learning models with attention mechanisms have shown impressive performance in hyperspectral (HS) pansharpening. However, most of these existing models follow an early/late-fusion strategy and do not take full advantage of the hierarchical features. In this paper, we specifically design a novel end-to-end multi-level feature fusion network with cross-layer guided attention for HS pansharpening, termed as HP-MFFN, which allows the network to extract the hierarchical features level by level. Specifically, as the different levels of the network have different receptive fields and contain different details, the hierarchical features extracted from the HS image and panchromatic (PAN) image are refined by the cross-layer guided attention fusion module (called CLGAF) to yield more effective spatial-spectral features with fine details and rich semantics. The experimental results conducted on widely-used datasets demonstrate that HP-MFFN provides high-quality pansharpened HS images in terms of perceptually and quantitatively.

Multi-level spatial details cross-extraction and injection network for hyperspectral pansharpening.

Hyperspectral (HS) pansharpening, which fuses the HS image with a high spatial resolution panchromatic (PAN) image, provides a good solution to overcome the limitation of HS imaging devices. However, most existing convolutional neural network (CNN)-based methods are hard to understand and lack interpretability due to the black-box design. In this Letter, we propose a multi-level spatial details cross-extraction and injection network (MSCIN) for HS pansharpening, which introduces the mature multi-resolution analysis (MRA) technology to the neural network. Following the general idea of MRA, the proposed MSCIN divides the pansharpening process into details extraction and details injection, in which the missing details and the injection gains are estimated by two specifically designed interpretable sub-networks. Experimental results on two widely used datasets demonstrate the superiority of the proposed method.