High-resolution Panchromatic Image Research Articles

Evaluation of Multi-Source High-Resolution Remote Sensing Image Fusion in Aquaculture Areas

Image fusion of satellite sensors can generate a high-resolution multi-spectral image from inputs of a high spatial resolution panchromatic image and a low spatial resolution multi-spectral image for feature extraction and target recognition, such as enclosure seines and floating rafts. However, there is currently no clear and definite method of image fusion for different aquaculture areas distribution extraction from high-resolution satellite images. This study uses three types of high-resolution remote sensing images, GF-1 (Gaofen-1), GF-2 (Gaofen-2), and WV-2 (WorldView-2), covering the raft and enclosure seines aquacultures in the Xiangshan Bay, China, to evaluate panchromatic and multispectral image fusion techniques to determine which is the best. This study applied PCA (principal component analysis), GS (Gram-Schmidt), and NNDiffuse (nearest neighbor diffusion) algorithms to panchromatic and multispectral images fusion of GF-1, GF-2, and WV-2. Two quantitative methods are used to evaluate the fusion effect. The first used seven statistical parameters, including gray mean value, standard deviation, information entropy, average gradient, correlation coefficient, deviation index, and spectral distortion. The second is the CQmax index. Comparing the evaluation results by these seven common statistical indicators with the results of the image fusion evaluation by index CQmax, the results prove that the CQmax index can be applied to the evaluation of image fusion effects in different aquaculture areas. For the floating raft cultured area, the conclusion is consentaneous; NNDiffuse was also optimal for GF-1 and GF-2 data, and PCA was optimal for WV-2 data. For the enclosure seines culture area, the conclusion of quantitative evaluations is not consistent and it shows that there is no definite good method that can be applied to all areas; therefore, careful evaluation and selection of the best applicable image fusion method are required according to the study area and sensor images.

Hyperspectral Pansharpening With Adaptive Feature Modulation-Based Detail Injection Network

Recently, deep learning-based methodologies have attained unprecedented performance in hyperspectral (HS) pansharpening, which aims to improve the spatial quality of HS images (HSIs) by making use of details extracted from the high-resolution panchromatic (HR-PAN) image. However, it remains challenging to incorporate the details into the pansharpened image effectively, while alleviating the spectral distortion simultaneously. To tackle this problem, in this article, we propose an adaptive feature modulation-based detail injection network (AFM-DIN) for HS pansharpening, which mainly consists of four phases: high-frequency details generation of the HR-PAN image, multiscale feature extraction of the upsampled HSI, AFM-based detail injection and reconstruction of the HR-HSI. First, a novel octave convolution unit is employed to decompose the HR-PAN image into high and low frequencies, and then merge the high-frequency features together to generate the comprehensive PAN-details. Second, the spatial and spectral separable 3D convolution units with multiple kernel sizes are designed to extract multiscale features of the upsampled HSI in a computationally efficient manner. Subsequently, by taking the critical PAN-details as prior, the proposed AFM module is able to not only incorporate the detail information effectively, but also adjust the injected details adaptively to ensure the spectral fidelity. Finally, the anticipated HR-HSI is obtained through adding the upsampled HSI to the predicted HSI-details reconstructed from informative modulated features. Extensive comparison experiments with several state-of-the-arts conducted on simulated and real HS data sets demonstrate that our proposed AFM-DIN can achieve superior pansharpening accuracy in both spatial and spectral aspects.

Detail Injection-Based Convolutional Auto-Encoder for Pansharpening

The purpose of pansharpening is to generate high-resolution multispectral (MS) images using both low-resolution MS images and high-resolution panchromatic images. Traditional remote sensing image fusion algorithms can be simplified to a unified detail injection (Di) context that treats the injected MS details as panchromatic-detail and integration with injection gain. The injected details are developed from traditional fusion strategies with clear physical interpretation and facilitate fast convergence of deep learning models for high-quality image fusion. The excellent ability of convolutional autoencoder (CAE) networks to retain image information enables its application to remote sensing image fusion. In this paper, a fusion method Di-based CAE (DiCAE) based on Di and CAE is proposed. DiCAE method is based on Di as the theoretical foundation and CAE network as the core of the algorithm. In addition, our method is evaluated through experiments on different satellite datasets, and the fusion results obtained by DiCAE have better objective evaluation metrics and better visual results compared to other state-of-the-art methods.

Effective Pan-Sharpening With Transformer and Invertible Neural Network

In remote sensing imaging systems, pan-sharpening is an important technique to obtain high-resolution multispectral images from a high-resolution panchromatic image and its corresponding low-resolution multispectral image. Due to the powerful learning capability of convolution neural networks (CNNs), CNN-based methods have dominated this field. However, due to the limitation of the convolution operator, long-range spatial features are often not accurately obtained, thus limiting the overall performance. To this end, we propose a novel and effective method by exploiting a customized transformer architecture and information-lossless invertible neural module for long-range dependencies modeling and effective feature fusion in this article. Specifically, the customized transformer formulates the panchromatic (PAN) and multispectral (MS) features as queries and keys to encourage joint feature learning across two modalities, while the designed invertible neural module enables effective feature fusion to generate the expected pan-sharpened results. To the best of our knowledge, this is the first attempt to introduce a transformer and a invertible neural network into the pan-sharpening field. Extensive experiments over different kinds of satellite datasets demonstrate that our method outperforms state-of-the-art algorithms both visually and quantitatively with fewer parameters and flops. Furthermore, the ablation experiments also prove the effectiveness of the proposed customized long-range transformer and effective invertible neural feature fusion module for pan-sharpening.

PSCSC-Net: A Deep Coupled Convolutional Sparse Coding Network for Pansharpening

Given a low-resolution multispectral (MS) image and a high-resolution panchromatic image, the task of pansharpening is to generate a high-resolution MS image. Deep learning (DL)-based methods receive extensive attention recently. Different from the existing DL-based methods, this article proposes a novel deep neural network for pansharpening inspired by the learned iterative soft thresholding algorithm. First, a coupled convolutional sparse coding-based pansharpening (PSCSC) model and related traditional optimization algorithm are proposed. Then, following the procedures of traditional algorithm for solving PSCSC, an interpretable end-to-end deep pansharpening network is developed using a deep unfolding strategy. The designed deep architecture can also be understood in the view of details injection (DI)-based scheme. This work offers a solution that integrates the DL-, DI-, and variational optimization-based schemes into a framework. The experimental results on the reduced- and full-scale datasets demonstrate that the proposed deep pansharpening network outperforms popular traditional methods and some current DL-based methods.

Multistage Dual-Attention Guided Fusion Network for Hyperspectral Pansharpening

Deep learning, especially the convolutional neural network, has been widely applied to solve the hyperspectral pansharpening problem. However, most do not explore the intraimage characteristics and the interimage correlation concurrently due to the limited representation ability of the networks, which may lead to insufficient fusion of valuable information encoded in the high-resolution panchromatic images (HR-PANs) and low-resolution hyperspectral images (LR-HSIs). To cope with this problem, we develop a hyperspectral pansharpening method called multistage dual-attention guided fusion network (MDA-Net) to fully extract the important information and accurately fuse them. It employs a three-stream structure, which enables the network to incorporate the intrinsic characteristics of each input and correlation among them simultaneously. In order to combine as much information as possible, we merge the features extracted from three streams in multiple stages, where a dual-attention guided fusion block (DAFB) with spectral and spatial attention mechanisms is utilized to fuse the features efficiently. It identifies the useful components in both spatial and spectral domains, which are beneficial to improving the fusion accuracy. Moreover, we design a multiscale residual dense block (MRDB) to extract dense and hierarchical features, which improves the representation power of the network. Experiments are conducted on both real and simulated datasets. The evaluation results validate the superiority of the MDA-Net.

Notice of Removal: P3Net: Pansharpening via Pyramidal Detail Injection With Deep Physical Constraints

Pansharpening is an image fusion process aiming to generate high-resolution multispectral (HRMS) images from a pair of low-resolution multispectral (LRMS) images and a high-resolution PAN image. It is a fundamental and significant task for the widespread use of remote sensing images. This paper proposes a new residual learning-based multispectral pansharpening network constrained by two deep physical models, collectively termed as P3Net. It mainly consists of the mainstream PDFNet and the other two auxiliary physical models, M2PNet and H2LNet. Unlike the existing methods of processing only one image scale, the proposed PDFNet fully extracts the spatial details from the multi-level image pyramid with decreasing spatial scales. Then, the spatial information is injected into the upsampled LRMS image. Since the pan-sharpened result should be consistent with the observed inputs under the physics models, we learn deep pansharpening physics models to reflect the inverse relationships. In detail, we propose the lightweight M2PNet and H2LNet to represent the latent non-linear mappings from the HRMS image to the panchromatic (PAN) image and the LRMS image, respectively. The two pre-trained physics models are frozen and guide the training of the PDFNet, so as to drive clear physical interpretability and further suppress the spectral and spatial distortions. The comparative experiments with the existing state-of-the-art pansharpening methods on QuickBird, GaoFen, and WorldView test sets demonstrate the superiority of the proposed method in terms of both quantitative metrics and subjective visual effects. The codes are available at https://github.com/KSJhon/PyramidPanWithPhysics.

Integrated Fusion for Panchromatic, Multispectral, Hyperspectral Remote Sensing Images With Different Swath Widths

Zi Yuan (ZY)-1 02D satellite simultaneously provides the low spatial resolution (LR) and narrow swath-width hyperspectral (HS) image, the moderate spatial resolution (MR) multispectral (MS) image with a wider swath width, and the high spatial resolution (HR) panchromatic (PAN) image with the same wide swath width to the MR MS. How to comprehensively integrate their complementary advantages to obtain the wide swath-width and high-fidelity HR HS image is interesting but challenging. In this paper, we propose an integrated fusion method for the HR PAN, MR MS, and LR HS images with different swath widths, to generate the optimal wide swath-width HR HS image. The proposed method is based on the encoder-decoder learning framework. In the proposed fusion framework, a novel multi-branch encoder structure with an enhanced HS-encoder module and the multilevel spatial-spectral aggregation block is designed, by considering the difference in the spatial and spectral resolution among the multi-sensor images. The experiments on synthetic and real datasets from both qualitative and quantitative aspects demonstrated the competitive performance of the proposed method.

Detail-Injection-Model-Inspired Deep Fusion Network for Pansharpening

Pansharpening is an image fusion procedure, which aims to produce a high spatial resolution multispectral image by combining a low spatial resolution multispectral image and a high spatial resolution panchromatic image. The most popular and successful paradigm for pansharpening is the framework known as detail injection, while it cannot fully exploit complex and non-linear complementary features of both images. In this paper, we propose a detail injection model inspired deep fusion network for pansharpening (DIM-FuNet). Firstly, by treating pansharpening as a complicated and non-linear details learning and injection problem, we establish a unified optimizing detail-injection model with triple detail fidelity terms: 1) a band-dependent spatial detail fidelity term, 2) a local detail fidelity term and 3) a complicated details synthesis term. Secondly, the model is optimized via the iterative gradient descent and unfolded into a deep convolutional neural network. Subsequently, the unrolling network has triple branches, in which, a point-wise convolutional sub-network, a depth-wise convolutional sub-network are corresponding to the former two detail constrained terms, and an adaptive weighted reconstruction module with a fusion sub-network to aggregate details of two branches and synthesis the final complicated details. Finally, the deep unrolling network is trained in end-to-end manners. Different from traditional deep fusion networks, the architecture design of DIM-FuNet is guided by the optimizing model and thus promotes better interpretability. Experimental results on reduced and full-resolution demonstrate the effectiveness of the proposed DIM-FuNet which achieves the best performance compared with the state-of-the-art pansharpening method.

No Reference Pansharpened Image Quality Assessment Through Deep Feature Similarity

Pansharpening refers to the process of enhancing the spatial resolution of a multispectral image with the help of a high spatial resolution panchromatic image. Quality assessment (QA) of pansharpened images helps provide a formal framework for the analysis and design of pansharpening methods and is thus extremely important. However, lack of availability of a reference multispectral image makes QA of pansharpening algorithms a challenging task. Given the popular use of QA algorithms that use a reference, this paper focuses on predicting the quality under a &#x2018;no-reference&#x0027; (NR) setting. Specifically, a learning based NR pansharpened image quality assessment (IQA) approach is adopted to predict state of the art reference based measures such as <inline-formula><tex-math notation="LaTeX">$Q2^{n}$</tex-math></inline-formula> and SAM without the need of a reference. We design an end-to-end deep pansharpening IQA network to compute the similarity of deep features fused from the panchromatic and input low resolution multispectral with similar features extracted from the given pansharpened image. To train and test our learning based approach, we create a large corpus of pansharpened images belonging to different satellites and thematic scenes by applying different pansharpening algorithms. Our experiments demonstrate that our NR pansharpened IQA algorithm achieves excellent performance and generalizes well across different satellites and resolutions. The source code and the dataset will be made available at <uri>https://github.com/neerajbadal/DPIQA</uri>.

Multiresolution Analysis Based on Dual-Scale Regression for Pansharpening

Pansharpening technique is used to merge the original multispectral image (MS) with a high spatial resolution panchromatic image (PAN). Due to its robustness, the multiresolution analysis (MRA) is an important part of pansharpening. The scale regression model is effective for improving MRA. However, the existing MRA based on scale regression results into single-scale regression information, thus affecting the final pansharpening result. To address this problem, in this work, we propose a dual-scale regression-based MRA for pansharpening. First, we establish a scale regression-based model. Then, this model is improved using a high-pass modulation (HPM) injection scheme. Finally, the dual-scale information is added to the scale regression to construct the dual-scale regression for obtaining the final pansharpening result. We perform experiments using five datasets. The results show that the proposed method obtains a better pansharpening result as compared to various state-of-the-art MRA methods. In addition, the quantitative and qualitative analysis of the results shows that the proposed method achieves appropriate spatial and spectral resolution fusion. Therefore, it has a great potential in pansharpening technique.

A New Context-Aware Details Injection Fidelity With Adaptive Coefficients Estimation for Variational Pansharpening

Pansharpening is related to the fusion of a low spatial resolution multispectral (MS) image retaining an abundant spectral content and a high spatial resolution panchromatic (PAN) image to obtain a product with both the abundant spectral content of the former and the high spatial resolution of the latter. Many previous studies are only focused on the global or local relationship between the PAN image and the corresponding high-resolution multispectral (HRMS) image. However, we found that the relationship between PAN and HRMS images in the gradient domain can be better explored through the image context. In this article, we propose context-aware details injection fidelity (CDIF) with adaptive coefficients estimation, which can fully explore the complicated relationship between the PAN image and the HRMS image in the gradient domain. More specifically, we apply a clustering method to divide the pixels of an image into different context-based regions. Afterward, the adaptive coefficients are estimated by using a regression-based method for each region. The CDIF is effective in extracting the main features from the two inputs to be fused. In addition, we integrate the CDIF with a conventional fidelity term and a total variation regularization to formulate a novel variational pansharpening model that is solved by designing an algorithm based on the alternating direction method of multiplier (ADMM) framework. Qualitative and quantitative assessments on different datasets support the effectiveness and robustness of the proposed method. The code is available at <uri>https://github.com/liangjiandeng/CDIF</uri>.

A Dual-Branch Detail Extraction Network for Hyperspectral Pansharpening

Hyperspectral (HS) pansharpening aims at creating a high-resolution hyperspectral (HR-HS) image by integrating a high spatial resolution panchromatic (HR-PAN) image with a low-resolution hyperspectral (LR-HS) image. It is an important preprocessing procedure in many remote sensing tasks. Most of the existing pansharpening methods train a specific convolutional neural network (CNN) model for each type of dataset with the same number of spectral bands. The main contribution of this study is to propose a new dual-branch detail extraction pansharpening network (called DBDENet) that can sharpen HS images with any number of spectral bands using a single pre-trained model by fine-tuning the parameters of a small module in the network. Specifically, DBDENet extracts spatial details from LR-HS and HR-PAN images by two bidirectional branches of the dual-branch detail extraction network level by level. For each level, the spatial details captured from the HR-PAN and those of the LR-HS images are fused by a spatial cross attention fusion module (SCAFM). The spatial details fused by the last SCAFM module are injected into the upsampled HS image to obtain an HR-HS image. Experimental results prove to show the proposed DBDENet is superior to other widely accepted state-of-the-art methods in terms of objective indicators and visual appearance.

PanCSC-Net: A Model-Driven Deep Unfolding Method for Pansharpening

Recently, deep learning (DL) approaches have been widely applied to the pansharpening problem, which is defined as fusing a low-resolution multispectral (LRMS) image with a high-resolution panchromatic (PAN) image to obtain a high-resolution multispectral (HRMS) image. However, most DL-based methods handle this task by designing black-box network architectures to model the mapping relationship from LRMS and PAN to HRMS. These network architectures always lack sufficient interpretability, which limits their further performance improvements. To address this issue, we adopt the model-driven method to design an interpretable deep network structure for pansharpening. First, we present a new pansharpening model using the convolutional sparse coding (CSC), which is quite different from the current pansharpening frameworks. Second, an alternative algorithm is developed to optimize this model. This algorithm is further unfolded to a network, where each network module corresponds to a specific operation of the iterative algorithm. Therefore, the proposed network has clear physical interpretations, and all the learnable modules can be automatically learned in an end-to-end way from the given dataset. Experimental results on some benchmark datasets show that our network performs better than other advanced methods both quantitatively and qualitatively.

Progressive Pan-Sharpening via Cross-Scale Collaboration Networks

Pan-sharpening aims to produce a high-quality image by fusing a low-resolution multispectral (MS) image and a high-resolution panchromatic (PAN) image. Although deep learning-based methods have dominated the pan-sharpening, they fail to fully utilize spatial and spectral information from the cross-scale perspective. In this work, we propose a novel cross-scale collaboration network to achieve accurate pan-sharpening. Specifically, we first design a progressive framework in a pyramid fashion to achieve a gradual pan-sharpening process, which consists of several subnetworks to handle specific pyramid levels. Then, in each subnetwork, we deploy two cross-scale attention modules to respectively capture global and local spatial interactions from MS and PAN images. To better utilize spectral information from different pyramid levels, we further deploy a fusion module between subnetworks to extract cross-scale representations. Through the above-mentioned cross-scale collaboration, our network can fully consider the cross-scale nature of MS and PAN images to better adapt to this specific task. Extensive experiments confirmed that our method outperforms several state-of-the-art methods both qualitatively and quantitatively.

A Deep-Shallow Fusion Network With Multidetail Extractor and Spectral Attention for Hyperspectral Pansharpening

Hyperspectral (HS) pansharpening aims at fusing a low-resolution HS (LRHS) image with a high-resolution panchromatic (PAN) image to obtain a hyperspectral image with both higher spectral and spatial resolutions. However, existing HS pansharpening algorithms are mainly based on multispectral (MS) pansharpening approaches, which cannot perfectly restore much spectral information and more high-frequency spatial details in the continuous spectral bands and much broader spectral range, leading to spectral distortion and spatial blur. In this paper, we develop a new hyperspectral pansharpening network architecture (called Hyper-DSNet) to fully preserve latent spatial details and spectral fidelity via a deep-shallow fusion structure with multi-detail extractor and spectral attention. Specifically, the proposed architecture mainly consists of three parts. First, to solve the problem of spatial ambiguity and exploit the potential information, five types of high-pass filter templates are used to fully extract the spatial details of the PAN image, constructing a so-called multi-detail extractor (MDE). Then, after passing a multi-scale convolution module, a deep-shallow fusion (DSF) structure, which reduces parameters by decreasing the number of output channels as the network goes deeper, is utilized sequentially. In final, a spectral attention (SA) module is conducted to preserve the spectrum for a wealth of spectral information of HS images. Visual and quantitative experiments on three commonly used simulated datasets and one full-resolution dataset demonstrate the effectiveness and robustness of the proposed Hyper-DSNet against the recent state-of-the-art hyperspectral pansharpening techniques. Ablation studies and discussions further verify our contributions, <i>e</i>.<i>g</i>., better spectral preservation and spatial detail recovery. The source code is available at <uri>https://github.com/liangjiandeng/Hyper-DSNet</uri>, and the related datasets can be found from <uri>https://github.com/liangjiandeng/HyperPanCollection</uri>.

Hyperspectral Intrinsic Image Decomposition With Enhanced Spatial Information

Hyperspectral intrinsic image decomposition (HyperIID) has been proven to be a very useful approach to reduce the spectral uncertainty in the remote sensing imaging process and improve the classification. In this article, a new HyperIID with enhanced spatial information, called ESI-IID, is proposed to overcome the deficiency of low spatial resolution in the existing HyperIID methods. With the aid of high-resolution (HR) panchromatic (PAN) image, the proposed method embeds the HR spatial information into the intrinsic decomposition model and enhances spatial details of the intrinsic component. The proposed ESI-IID introduces three constraints: 1) we make the constraint on spectral information to protect it from distortion during the spatial resolution enhancement process; 2) we add the constraint on spatial information to make sure that the details of edges will be well kept; and 3) based on the assumption that the reflectance component has a strong correlation in the local neighborhood, we add the self-constraint on reflectance component, in which the similarity matrix consists of two parts extracted from hyperspectral images and PAN image, respectively. Finally, we build a matrix energy function according to the aforementioned constraints and solve it by finding the minimum Frobenius norm iteratively. Both visual and quantitative experiments on simulated and real datasets demonstrate that the proposed method outperforms other alternative methods with high reliability.

Novel Cross-Resolution Feature-Level Fusion for Joint Classification of Multispectral and Panchromatic Remote Sensing Images

With the increasing availability and resolution of satellite sensor data, multispectral (MS) and panchromatic (PAN) images are the most popular data that are used in remote sensing among applications. This article proposes a novel cross-resolution hidden layer feature fusion (CRHFF) approach for joint classification of multiresolution MS and PAN images. In particular, shallow spectral and spatial features at a global scale are first extracted from an MS image. Then, deep cross-resolution hidden layer features extracted from MS and PAN are fused from patches at a local scale according to an autoencoder (AE)-like deep network. Finally, the selected multiresolution hidden layer features are classified in a supervised manner. By taking advantage of integrated shallow-to-deep and global-to-local features from the high-resolution MS and PAN images, the cross-resolution latent information can be extracted and fused in order to better model imaged objects from the multimodal representation and finally increase the classification accuracy. Experimental results obtained on three real multiresolution datasets covering complex urban scenarios confirm the effectiveness of the proposed approach in terms of higher accuracy and robustness with respect to literature methods.

A Model-Inspired Approach With Transformers for Hyperspectral Pansharpening

Hyperspectral pansharpening is a method of integrating a low-resolution hyperspectral image with a high-resolution panchromatic image to produce a high-resolution hyperspectral image. In recent years, a number of hyperspectral pansharpening methods have been developed by using convolutional neural networks. However, these methods only consider local information due to limitations on the size of convolution kernels in the convolution operation. In this paper, we design a model-inspired approach with transformers for hyperspectral pansharpening. Due to the representation ability of transformers and the algorithmic explanatory ability of optimization models, our method is able to explore intrinsic relationships at a global scale on both spectral and spatial features. First, we formulate an optimization model consisting of a fidelity term and a regularization term. Then, this optimization model is solved by a half quadratic splitting algorithm, and thus divided into two sub-optimization problems, the fidelity and regularization problems. Finally, the algorithm is implemented by a transformer-based deep structure. Specifically, the fidelity problem is solved by a gradient descent algorithm further, and then implemented through a convolutional network. The regularization problem is depicted by an approximation operation and implemented via a transformer network. The experimental results on different satellite datasets demonstrate the effectiveness of the proposed method.

Spatial Dynamic Selection Network for Remote-Sensing Image Fusion

Nowadays, high-resolution images with rich spectral information are necessary for earth observation. Remote-sensing image fusion is an effective method to provide high-resolution multispectral images, which are obtained by fusing high-resolution panchromatic images and low-resolution multispectral images. However, existing methods mostly use the same network for image feature extraction, without considering the differences among different pixels, resulting in that the extracted features are not accurate enough. This letter proposes a spatial dynamic selection network for remote-sensing image fusion. A dynamic feature extraction module composed of multiple spatial dynamic blocks (SDBs) and cross-scale context connection blocks (CSCBs) is designed. The SDB can extract image features according to the input by different networks, and realize dynamic selection of pixel features. Since the spatial structure and spectral characteristic of each pixel are different, two complementary branches are designed in the SDB to extract different features, which improves the capability of feature extraction. Multiscale network structure is designed to obtain more abundant information and the CSCB is used to integrate the information of different scales. Experimental results on GeoEye-1 and WorldView-3 datasets demonstrate the superiority of the proposed method.