Panchromatic Image Fusion Research Articles

Multimodal medical image fusion based on joint bilateral filter and local gradient energy

• An effective, fast and noise robust multimodal medical image fusion method is proposed. An image is modeled as a superposition of the structure layer and the energy layer decomposed by joint filter, then a novel local gradient energy operator is proposed to fuse the structure layer and the abs-max rule to fuse the energy layer. • 118 co-registered pairs of medical images cover five different categories of medical image fusion problems (MR-T1/MR-T2, CT/MR, MR-Gad/MR-T2, MR/PET, and MR/SPECT) are tested. To the best of our knowledge, this is the most comprehensive experiment both in quantity and classes for medical image fusion. • The merits of noniterative, local, and without multiscale decomposition or reconstruction make the proposed method easy to implementation and achieve high computational efficiency. • Our method can be effectively extended to other image fusion problems, such as multifocus image fusion, infrared and visible image fusion, millimeter-wave and visible image fusion, panchromatic and multispectral image fusion. As a powerful assistance technique for biomedical diagnosis, multimodal medical image fusion has emerged as a hot topic in recent years. Unfortunately, the trade-off among fusion performance, time consumption and noise robustness for many medical image fusion algorithms remains an enormous challenge. In this paper, an effective, fast and robust medical image fusion method is proposed. A two-layer decomposition scheme is introduced by the joint bilateral filter, the energy layer containing rich intensity information, and the structure layer capturing ample details. Then a novel local gradient energy operator based on the structure tensor and neighbor energy is proposed to fuse the structure layer and the l 1 -max rule is introduced to fuse the energy layer. A total of 118 co-registered pairs of medical images covering five different categories of medical image fusion problems are tested in experiments. Seven latest representative medical image fusion methods are compared, and six representative quality evaluation metrics with complementary characteristics are fully employed to objectively evaluate the fused results. Extensive experimental results demonstrate that the proposed method yields better performance than some state-of-the-art methods in both visual quality and quantitative evaluation, and achieves nearly real-time computational efficiency and robustness to noise.

Remote sensing image fusion with multistream deep ResCNN

Remote sensing image pan-sharpening is the fusion of multispectral (MS) image and panchromatic (PAN) image into fused image with MS and high-resolution information features. In recent years, deep learning technology has been widely used in abounding fields such as text and image recognition. Inspired by this, we proposed a multistream deep ResCNN (RIFRCNN)-based fusion method for remote sensing images. First, we put forward automatic learning matching module (ALMM). The utilization of ALMM can realize dynamic optimization matching of MS and PAN images with different resolutions. Second, we build feature fusion residual blocks suitable for RIFRCN to optimize network parameters. Finally, we input MS image and PAN image with different resolution and complete the fusion task through ALMM, fusion, and reconstruction module. Experiments on WorldView and QuickBird datasets manifest that the proposed RIFRCNN can effectively fuse MS and PAN images, and the fusion images generated in the experiments of true and false color images show better results than the state-of-the-art technology.

MSSL: Hyperspectral and Panchromatic Images Fusion via Multiresolution Spatial–Spectral Feature Learning Networks

The fusion of hyperspectral (HS) and panchromatic (PAN) images aims to generate a fused HS image that combines spectral information of the HS image with spatial information of the PAN image. In this article, we propose a multiresolution spatial–spectral feature learning (MSSL) framework for fusing HS and PAN images. The proposed MSSL transforms the existing deep and complex network into several simple and shallow subnetworks to simplify the feature learning process. MSSL upsamples the HS image while downsamples the PAN image and designs multiresolution 3-D convolutional autoencoder (CAEs) networks with a spectral constraint to learn complete spatial–spectral features of the HS image. MSSL designs multiresolution 2-D CAEs with spatial constraint to extract spatial features of the PAN image, with a low computational cost. In order to effectively generate the pansharpened HS image with high spatial and spectral fidelity, a multiresolution residual network is presented to reconstruct the HS image from the extracted spatial–spectral features. Extensive experiments are conducted on three widely used remote sensing data sets in comparison with state-of-the-art HS image fusion methods, demonstrating the superiority of the proposed MSSL method. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Jiahuiqu/MSSL</uri> .

Dual-Collaborative Fusion Model for Multispectral and Panchromatic Image Fusion

The aim of multispectral (MS) and panchromatic (PAN) image fusion is to obtain an MS image that has high resolution in both spectral and spatial domains. During the fusion process, there are two important issues, i.e., spectral information preservation and spatial information enhancement. In this article, we propose a dual-collaborative fusion model that considers not only the spectral correlation collaboration but also the spatial-spectral collaboration. First, the features of PAN and MS images are extracted by a shared feature embedding network. Then, in order to enhance the spatial details, the PAN features are decomposed into four subbands, and the collaborative relationships among subbands are fully explored to refine the features. After the refinement of the subbands, the high-frequency components are directly taken as the inputs of the reconstruction network, while the low-frequency components are transformed by the guidance generation network to accomplish the spatial-spectral collaboration and also make preparations for the spectral adjustment. To explore the spectral correlation collaboration, a novel graph convolutional network is designed for the modulation of intraspectral relationships. Finally, the adjusted MS features are combined with the high-frequency components of PAN features to reconstruct the high-resolution MS image. Experimental results show that the proposed method outperforms traditional state-of-the-art pan-sharpening methods as well as the available deep learning-based ones.

Ship Classification by the Fusion of Panchromatic Image and Multi-spectral Image Based on Pseudo Siamese LightweightNetwork

The current rapid development of the remote sensing satellite industry provides a large amount of image data for ship classification tasks. Aiming at the problem of insufficient feature extraction of single source image, this paper designs a lightweight ship classification model based on the fusion of panchromatic image and multispectral image of pseudo Siamese network to extract image features more fully. First, establish a multi-source remote sensing image ship target classification dataset MPFS (MS and PAN Ship image Fusion Classification Dataset); secondly, send panchromatic images and multispectral images to the network through different convolutional layers, thendesign a multi-level feature extraction network for panchromatic images and an adaptive feature extraction network for spectral imagesrespectively; finally, concatenate the features along the channel dimension and send them to the classification network.

Spectral and spatial information improvement multispectral image based on recursive full-scale estimation

ABSTRACT Pansharpening refers to the fusion of panchromatic (Pan) image and multispectral (MS) image, which are acquired from the same scene. The output of the process is a high-spatial-resolution MS image. However, in most cases, spectral or spatial distortion occurs in the local region. To solve this problem, this paper introduces a detailed injection model, where estimates the model parameters recursively at the full-scale. The proposed method estimates detailed injection coefficients by finding minimum variance-unbiased (MVU) estimator. The superiority of the proposed method is verified via conducting several experiments on the five satellite datasets and comparing the results quantitatively and visually with several existing methods.

Hyperspectral Image Fusion and Multitemporal Image Fusion by Joint Sparsity

Different image fusion systems have been developed to deal with the massive amounts of image data for different applications, such as remote sensing, computer vision, and environment monitoring. However, the generalizability and versatility of these fusion systems remain unknown. This article proposes an efficient regularization framework to achieve different kinds of fusion tasks accounting for the spatiospectral and spatiotemporal variabilities of the fusion process. A joint minimization functional is developed by taking an advantage of a composite regularizer for enforcing joint sparsity in the gradient domain and the frame domain. The proposed composite regularizer is composed of the Hessian Schatten-norm regularization and contourlet-based regularization terms. The resulting problems are solved by the alternating direction method of multipliers (ADMM). The effectiveness of the proposed method is validated in a variety of image fusion experiments: 1) hyperspectral (HS) and panchromatic image fusion; 2) HS and multispectral image fusion; 3) multitemporal image fusion (MIF); and 4) multi-image deblurring. Results show promising performance compared with state-of-the-art fusion methods.

The PAN and MS image fusion algorithm based on adaptive guided filtering and gradient information regulation

In recent years, with the improvement in the accuracy of remote sensing image classification and target recognition, the feature level fusion technology of remote sensing images has attracted much attention and become a research hotspot. However, this kind of fusion technology is not as mature as pixel-level fusion technology, and there are still many problems to be solved. This paper proposes a multi-spectral (MS) and panchromatic (PAN) image fusion algorithm based on adaptive textural feature extraction and information injection regulation. The fusion algorithm includes two stages. The first stage extracts the textural details of high-resolution PAN images. In this stage, based on the sensitivity of the remote sensing images to the gray-level co-occurrence matrix (GLCM), an adaptive guided filter (AGIF) scheme for remote sensing images based on the GLCM is proposed. The feature information of the textures and details of the PAN image was fully extracted. The second stage injects the extracted feature information of the PAN image into an MS image. In this stage, a decision map based on the MS image gradient domain and a weighted matrix based on the gradient entropy measure were proposed in order to, respectively, realize the adaptability of the feature injection location selection and regulate the intensity of the injected information to the MS image. This ensures the rationality of the injection of the textural information and avoids noise, patches and other information interference. The proposed algorithm has the advantages of fully extracting the textural features of high-resolution PAN images, adaptively adjusting the injection position and intensity when injecting the feature information into an MS image, and providing the fused image with clear features. On the premise of effectively maintaining the spectral information quality, the spatial resolution of the fused image is improved. A large number of simulation experiments verify the effectiveness of the proposed method.

HPGAN: Hyperspectral Pansharpening Using 3-D Generative Adversarial Networks

Hyperspectral (HS) pansharpening, as a special case of the superresolution (SR) problem, is to obtain a high-resolution (HR) image from the fusion of an HR panchromatic (PAN) image and a low-resolution (LR) HS image. Though HS pansharpening based on deep learning has gained rapid development in recent years, it is still a challenging task because of the following requirements: 1) a unique model with the goal of fusing two images with different dimensions should enhance spatial resolution while preserving spectral information; 2) all the parameters should be adaptively trained without manual adjustment; and 3) a model with good generalization should overcome the sensitivity to different sensor data in reasonable computational complexity. To meet such requirements, we propose a unique HS pansharpening framework based on a 3-D generative adversarial network (HPGAN) in this article. The HPGAN induces the 3-D spectral-spatial generator network to reconstruct the HR HS image from the newly constructed 3-D PAN cube and the LR HS image. It searches for an optimal HR HS image by successive adversarial learning to fool the introduced PAN discriminator network. The loss function is specifically designed to comprehensively consider global constraint, spectral constraint, and spatial constraint. Besides, the proposed 3-D training in the high-frequency domain reduces the sensitivity to different sensor data and extends the generalization of HPGAN. Experimental results on data sets captured by different sensors illustrate that the proposed method can successfully enhance spatial resolution and preserve spectral information.

Hyperspectral and Panchromatic Image Fusion via Adaptive Tensor and Multi-Scale Retinex Algorithm

Fusion of panchromatic image (PANI) and hyperspectral image (HSI) to obtain an output image with high spatial and spectral resolutions has received increasing interests recently. We propose a new image fusion method for HSI and PANI by combining adaptive tensor with a multi-scale Retinex algorithm in this paper. In the proposed method, an adaptive tensor based method is presented to effectively extract the structure information of HSI, and multi-scale Retinex algorithm is introduced to obtain the spatial and structure details of PANI. To integrate spatial structure information, a gradient-based weighted fusion strategy is proposed to combine spatial details of HSI and PANI. The integrated structure details are injected to generate the fused HSI. Experiments using both simulated and real remote sensing data sets demonstrated that the proposed fusion algorithm performs better than the state-of-the-art algorithms in visual inspection and objective assessment.

A New Variational Approach Based on Proximal Deep Injection and Gradient Intensity Similarity for Spatio-Spectral Image Fusion

Pansharpening is a very debated spatio-spectral fusion problem. It refers to the fusion of a high spatial resolution panchromatic image with a lower spatial but higher spectral resolution multispectral image in order to obtain an image with high resolution in both the domains. In this article, we propose a novel variational optimization-based (VO) approach to address this issue incorporating the outcome of a deep convolutional neural network (DCNN). This solution can take advantages of both the paradigms. On one hand, higher performance can be expected introducing machine learning (ML) methods based on the training by examples philosophy into VO approaches. On other hand, the combination of VO techniques with DCNNs can aid the generalization ability of these latter. In particular, we formulate a $\ell _2$-based proximal deep injection term to evaluate the distance between the DCNN outcome, and the desired high spatial resolution multispectral image. This represents the regularization term for our VO model. Furthermore, a new data fitting term measuring the spatial fidelity is proposed. Finally, the proposed convex VO problem is efficiently solved by exploiting the framework of the alternating direction method of multipliers (ADMM), thus guaranteeing the convergence of the algorithm. Extensive experiments both on simulated, and real datasets demonstrate that the proposed approach can outperform state-of-the-art spatio-spectral fusion methods, even showing a significant generalization ability. Please find the project page at https://liangjiandeng.github.io/Projects_Res/DMPIF_2020jstars.html.

Image fusion employing adaptive spectral-spatial gradient sparse regularization in UAV remote sensing

Unmanned aerial vehicle (UAV) remote sensing has been widely used in vegetation phenotypes and precision agriculture. The fusion of UAV multispectral and panchromatic images has considerable research value. For example, an accurate vegetation index can be obtained. However, large geometrical distortions are observed in UAV images, contributing to the insufficiency of existing fusion algorithms. Spectrum consistency, which indicates that changes in spectral direction are always a smooth function, is investigated in this paper to solve the above problem. Spatial adaptivity is also introduced to reduce spectral distortion in the fusion process. Based on the two aspects, a multispectral and panchromatic image fusion model employing adaptive spectral-spatial gradient sparse regularization is proposed for UAV remote sensing. The separable approximation and augmented Lagrangian methods are employed to optimize this model. In the experiments, the proposed method is firstly compared with other state-of-the-art fusion algorithms, and good performance is verified by UAV datasets in terms of visual effect and objective quality analysis. Secondly, the fusion algorithm is applied in the application of a vegetation phenotype. The experiments finally demonstrate that accurate vegetation indices can be generated by adopting the proposed algorithm. This finding proves the substantial research value of the proposed algorithm in UAV remote sensing.

Going Deeper with Densely Connected Convolutional Neural Networks for Multispectral Pansharpening

In recent years, convolutional neural networks (CNNs) have shown promising performance in the field of multispectral (MS) and panchromatic (PAN) image fusion (MS pansharpening). However, the small-scale data and the gradient vanishing problem have been preventing the existing CNN-based fusion approaches from leveraging deeper networks that potentially have better representation ability to characterize the complex nonlinear mapping relationship between the input (source) and the targeting (fused) images. In this paper, we introduce a very deep network with dense blocks and residual learning to tackle these problems. The proposed network takes advantage of dense connections in dense blocks that have connections for arbitrarily two convolution layers to facilitate gradient flow and implicit deep supervision during training. In addition, reusing feature maps can reduce the number of parameters, which is helpful for reducing overfitting that resulted from small-scale data. Residual learning is explored to reduce the difficulty for the model to generate the MS image with high spatial resolution. The proposed network is evaluated via experiments on three datasets, achieving competitive or superior performance, e.g. the spectral angle mapper (SAM) is decreased over 10% on GaoFen-2, when compared with other state-of-the-art methods.

A collaborative representation-based approximation method for remote sensing image fusion

ABSTRACTA collaborative representation-based approximation (CRA) for the fusion of low resolution multispectral (MS) image and high-resolution Panchromatic (Pan) image is proposed in this paper. To approximate each pixel of the high-resolution MS, the corresponding pixel in MS and the surrounding neighbours in its associated local window of Pan are contributed. Moreover, for increasing the spatial details in the final fused product, the spatial details of Pan are added to the fused image. The CRA method simply uses the spatial neighbours in the local regions and has no need to compose any appropriate dictionary for image representation. Moreover, it involves a -norm optimization problem which results in a closed-form solution. Various experiments visually and numerically show the superior performance of the proposed CRA method compared to several popular and recent pansharpening methods.

Fusion of panchromatic image with multi-spectral image using robust adaptive normalized convolution

This paper proposes a novel technique for reconstruction of a high resolution (HR) image by fussing irregularly sampled data of panchromatic image and multi-spectral image. The idea behind the fusion process is normalized convolution which uses the technique of approximating the local signals by projecting on to subspace. The normalized convolution used in this paper contains the polynomial basis functions similar to that of a Taylor series expansion. The window function used in the adaptive normalized convolution helps in improving the analysis by grouping more samples of same sense. This in turn helps to escalate the signal to noise ratio and curtail the diffusion across discontinuities. The necessity of multi-sensor data in required in many fields such as defense, remote-sensing, machine vision, medical imaging and the proposed method supplied promising results to such applications.

Image Fusion for High-Resolution Optical Satellites Based on Panchromatic Spectral Decomposition.

Ratio transformation methods are widely used for image fusion of high-resolution optical satellites. The premise for the use the ratio transformation is that there is a zero-bias linear relationship between the panchromatic band and the corresponding multi-spectral bands. However, there are bias terms and residual terms with large values in reality, depending on the sensors, the response spectral ranges, and the land-cover types. To address this problem, this paper proposes a panchromatic and multi-spectral image fusion method based on the panchromatic spectral decomposition (PSD). The low-resolution panchromatic and multi-spectral images are used to solve the proportionality coefficients, the bias coefficients, and the residual matrixes. These coefficients are substituted into the high-resolution panchromatic band and decompose it into the high-resolution multi-spectral bands. The experiments show that this method can make the fused image acquire high color fidelity and sharpness, it is robust to different sensors and features, and it can be applied to the panchromatic and multi-spectral fusion of high-resolution optical satellites.

Adaptive signal representation and multi-scale decomposition for panchromatic and multispectral image fusion

Abstract Multispectral images acquired by remote sensors although have a relative low spatial resolution, are valuable sources of spectral information. In contrast, panchromatic images with high spatial resolution have a lack of spectral features. Therefore, fusion of multispectral and panchromatic images called pansharpening provides an image that simultaneously has both of high spectral and spatial resolution. A hybrid pansharpening method that benefits the advantages of a multi-scale decomposition transform and free distribution model based methods is introduced in this work. The proposed method uses collaborative representation (CR) in addition to sparse representation (SR) for fusion of low pass components of multispectral and panchromatic images. An image consists of a background, which composes the majority of scene, and some targets, which have different spectral signatures with respect to background and occur with lower probability than it. In SR, just a few dictionary atoms contribute in signal representation while in CR all of dictionary atoms contribute for signal approximation. Therefore, SR is more appropriate for targets approximation while CR is more appropriate for background estimation. An anomaly score (AS) is computed for each image patch. Proportional to the calculated AS, a combination of SR and CR are adaptively used for approximation of image patch and generation of the fusion product. The proposed method is compared to some state-of-the-art pansharpening methods in terms of both the quality measures and visual analysis. The experiments are done on four datasets acquired by different sensors. The results show the superior performance of the proposed pansharpening method compared to different component substitution, multiresolution analysis, SR, and several hybrid methods.

Spatial frequency discrete wavelet transform image fusion technique for remote sensing applications

In Remote Sensing, fusion of Panchromatic (PAN) image and Multispectral (MS) image is an important technique. This paper incorporates a multiresolution image fusion algorithm based on the proposed Spatial Frequency DWT (SFDWT – Spatial Frequency Discrete Wavelet Transform) technique. In SFDWT technique, the low resolution MS image is resampled to the high resolution PAN image and fusion is done by injecting the spectral and spatial information’s present in PAN and MS images onto each other using their corresponding DWT coefficients by evaluating the spatial frequency (SF – Spatial Frequency). Evaluation is carried out using Pléiades Satellite images having a resolution ratio of 1:4. Fusion simulation is done in which pan sharpened reference images are available, indicate that the proposed SFDWT technique performs better than standard DWT methods and the IHS method in terms of different evaluation indexes including Structural similarity measure (SSIM), Errur Relative Globale Adimensionnelle de Synthese (ERGAS) etc. SFDWT produces high spectral and spatial quality fused images for remote sensing applications.

A Bayesian Procedure for Full-Resolution Quality Assessment of Pansharpened Products

Pansharpening regards the fusion of a high-spatial resolution panchromatic image with a low-spatial resolution multispectral image. One of the most debated topics about pansharpening is related to the quality assessment of fused products. Two main assessment procedures are usually exploited in the literature: the reduced resolution validation and the full-resolution (FR) validation. The former has the advantage to be accurate, but the hypothesis of invariance among scales has to be assumed. The latter overcomes this limitation but paying it with a lower accuracy. In this paper, we will focus on the FR assessment proposing an approach for estimating an overall quality index at FR by using multiscale FR measurements. The problem is recast into the sequential Bayesian framework exploiting a Kalman filter to find its solution. The proposed procedure for quality evaluation has been tested on four real data sets acquired by the Pleiades, the GeoEye-1, the WorldView-3, and the WorldView-4 sensors assessing the quality of 19 pansharpened methods. The proposed approach has demonstrated its superiority with respect to the benchmark consisting of state-of-the-art quality assessment procedures.

Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model

With the launch and rapid development of new satellites such as WorldView-3, the bands number of multi-spectral images from new satellites is greatly increased. However, the spectral matching between the panchromatic image and multi-spectral images is deteriorated with the existing image fusion methods. In this paper, a novel method based on the multi-channel deep model is proposed to fuse images for new satellites. The deep model is implemented by convolutional neural networks and trained on each band to reduce the impact of spectral range mismatch. The proposed method also preserves the detailed information in multi-spectral images, which is ignored by the traditional methods. It also effectively alleviates the inconvenience for obtaining the remote sensing images by the data augmentation processing. In addition, it reduces the randomness of manual setting parameters using the parameter self-learning. Visual and quantitative assessments of fusion results show that the proposed method clearly improves the fusion quality compared to the state-of-the-art methods.