Remote Sensing Image Fusion Method Research Articles

Remote sensing image fusion method based on depth feature extraction

Abstract Remote sensing image fusion involves combining data from various remote sensing sensors or merging multiple images acquired by the same sensor but in different spectral bands to create a superior quality composite image, high resolution, multi-band, and multi-temporal remote sensing image through certain algorithms and technologies. By utilizing fusion techniques, remote sensing images of varying bands, resolutions, and phases can be combined to enhance the accuracy of ground object information. This paper introduces a remote sensing image fusion method that leverages deep learning for depth feature extraction. Swin Transformer makes full use of the characteristics of the relationship between adjacent pixels, and adaptively extracts panchromatic image (PAN) and multispectral image (MS) features through two adaptive feature extraction channels. High-quality satellite data is employed to confirm the efficacy of the suggested approach, with the findings demonstrating strong performance in ERGAS, SCC, SSIM, and PSNR, and other evaluation indexes and has good spectral information fidelity and spatial information integration degree.

Multi-modal remote sensing image fusion method guided by local extremum maps-guided image filter

Multi-modal remote sensing image fusion method guided by local extremum maps-guided image filter

A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening

Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.

Triple disentangled network with dual attention for remote sensing image fusion

Remote sensing applications, such as detection and recognition of objects, need a large number of high spatial resolution (HR) images. As a feasible technique, image fusion is considered to produce HR images. However, most of remote sensing image fusion methods based on deep neural network (DNN) are mainly limited by the extraction of spatial and spectral features. These methods ignore the redundancy among these features, which causes spatial and spectral distortions in the fused image. In this paper, we propose a novel image fusion method based on a triple disentangled network (TDNet) with dual attention to reduce the redundancy in the extracted features. In the proposed method, it is assumed that the information in panchromatic (PAN) and low spatial resolution multispectral (LRMS) images can be encoded as the spatial, spectral, and common features. Specifically, we construct a triple-stream network to extract these features. To efficiently model the spatial and spectral information in PAN and LRMS images, local–global attention and interdependency attention are designed and integrated into the network. Then, the redundancy among these features is reduced by disentangled learning, in which these features are recombined to reconstruct the PAN and LRMS images. Besides, these features should complement each other. So, we utilize the maximal coding rate reduction to balance the redundancy and complementarity among them. Finally, all features are recombined to synthesize the high spatial resolution multispectral image. The experimental results demonstrate that the proposed TDNet has a superior performance in terms of qualitative and quantitative evaluations. The code link is https://github.com/RSMagneto/TDNet.

Improved GSA Method Based on Deep Convolutional Features

With the rapid development of remote sensing platforms, sensors and other technologies, remote sensing, as an important technical means of collecting information on land cover and its changes, plays an important role in land cover classification and dynamic monitoring. Due to the limitation of the imaging mechanism of in-orbit optical remote sensing sensors, it is difficult for to take into account both spatial resolution and spectral resolution. There is a need for complementary use of different sensor images and to improve the accuracy of feature interpretation. In this study, based on the Gram-Schmidt Adaptive (GSA) and its classical extension, a neural network-based multi-source remote sensing image fusion method is proposed by combining the U-Net coding and decoding structure with the Non-Local spatial attention mechanism. Using two sets of experimental datasets, three kinds of fusion evaluation metrics without reference and three kinds of fusion evaluation metrics with reference, the improved method and the existing fusion method are compared.

Remote Sensing Image Fusion Method Based on Progressive Cascaded Deep Residual Network

With the rapid development of deep learning in recent years, it has shown excellent performance in various image and video processing tasks. In addition, it also has a great role in promoting the spatio-temporal fusion of remote sensing images. The reconstructed image can give people a good visual experience. The invention relates to a remote sensing image fusion method based on a progressive cascade deep residual network and provides an end-to-end progressive cascade deep residual network model for remote sensing image fusion. The use of the MSE loss function may cause oversmoothing of the fused image, so a new joint loss function is defined to capture finer spatial information to improve the spatial resolution of the fused image. Resize-convolution is used to replace the transposed convolution to eliminate the checkerboard effect in the fused image caused by the transposed convolution. Through the experiments on the remote sensing image fusion simulation and real datasets of multiple satellites, the data results of the proposed algorithm are more than 5.25% better than those of the comparative algorithm in the average quantification. The calculation time and system resource occupation are also reduced, which has important theoretical significance and application value in the field of artificial intelligence and image processing. It will play a certain role in promoting the theoretical research and application of remote sensing image fusion.

Mun-GAN: A Multiscale Unsupervised Network for Remote Sensing Image Pansharpening

In remote sensing image fusion, pansharpening is a type of remote sensing image fusion method that aims to fuse panchromatic (PAN) images and multispectral (MS) images to produce high-resolution multispectral (HRMS) images. Deep learning based pansharpening technology offers a series of advanced unsupervised algorithms. However, there are several challenges: (1) The existing unsupervised pansharpening methods only consider the fusion of single-scale features; (2) for the fusion of MS and PAN image feature branches, the existing pansharpening methods are implemented directly by concatenation and summation, without paying attention to critical features or suppressing redundant features; (3) the semantic gap in the long skip connections of the network architecture will create unexpected results. In this paper, we design a multiscale unsupervised architecture based on generative adversarial networks (GANs) for remote sensing image pansharpening (Mun-GAN), which consists of a generator and two discriminators. The generator includes a multi-scale feature extractor (MFE), a self-adaptation weighted fusion (SWF) module, and a nest feature aggregation (NFA) module. First, the MFE is utilized to extract multiscale feature information from the input images and to then pass this information to the SWF module for adaptive weight fusion. Then, multiscale features are reconstructed by the NFA module to obtain HRMS images. The two discriminators are spectral and spatial discriminators used against the generator. Moreover, we design a hybrid loss function to aggregate the multiscale spectral and spatial feature information. Compared with other state-of-the-art methods using QuickBird, GaoFen-2 and WorldView-3 images, which demonstrate that the Mun-GAN yields better fusion results.

Remote sensing image fusion based on NSST and parameter adaptive PCNN

Remote sensing image fusion aims to enhance the spatial resolution of multispectral (MS) images by extracting spatial detail information from panchromatic (PAN) images. To preserve the spectral information of MS images and enhance the spatial resolution, an innovative remote sensing image fusion method is proposed here. The proposed method is based on an image matting model, the non-subsampled shear wave transform (NSST) and a parameter adaptive pulse coupled neural network (PA-PCNN). In the proposed method, an intensity component, I, is first obtained by adaptively weighted averaging over each band of the MS image. An image matting model is then applied. There, I is used as the α channel to estimate the foreground color, F, and the background color B. Then, the I and PAN images are decomposed into low and high-frequency coefficients by NSST. Two fusion rules are proposed here and applied to fuse the low and high-frequency coefficients, respectively. The fused low and high-frequency coefficients are inversely transformed to get a fused image, which is used as the α channel in the reconstruction of the image. Finally, the F, B and fusion images are combined to reconstruct the final high-resolution fused image. The experimental results show that the proposed approach achieves superior results in both subjective visual effects and objective assessment. It can enhance the spatial resolution and effectively retaining the spectral information of MS images.

Remote Sensing Image Fusion Based on Morphological Convolutional Neural Networks with Information Entropy for Optimal Scale.

Remote sensing image fusion is a fundamental issue in the field of remote sensing. In this paper, we propose a remote sensing image fusion method based on optimal scale morphological convolutional neural networks (CNN) using the principle of entropy from information theory. We use an attentional CNN to fuse the optimal cartoon and texture components of the original images to obtain a high-resolution multispectral image. We obtain the cartoon and texture components using sparse decomposition-morphological component analysis (MCA) with an optimal threshold value determined by calculating the information entropy of the fused image. In the sparse decomposition process, the local discrete cosine transform dictionary and the curvelet transform dictionary compose the MCA dictionary. We sparsely decompose the original remote sensing images into a texture component and a cartoon component at an optimal scale using the information entropy to control the dictionary parameter. Experimental results show that the remote sensing image fusion method proposed in this paper can effectively retain the information of the original image, improve the spatial resolution and spectral fidelity, and provide a new idea for image fusion from the perspective of multi-morphological deep learning.

Combined with Local Neighborhood Characteristics and Remote Sensing Image Fusion Method of C-BEMD

For the sake of ameliorate the high resolution recognition capacity building remote sensing images, a remote sensing image fusion method based on local neighborhood characteristics and C-BEMD is advanced. The building remote sensing image acquisition model and the building remote sensing image picture element edge feature detection model are designed. The wavelet multi-scale denoising method is used to suppress the fuzzy spread of picture element feature points between image residual units, extract the geometric feature points of image sequence, and process the building remote sensing image block by block. The global residual learning and message fusion of building remote sensing image are implemented. The local neighborhood feature matching method is used to reconstruct the building remote sensing image region. Combined with the C-BEMD empirical mode decomposition method, the building remote sensing image fusion and feature point matching in affine region are implemented, and the block image template matching method is used to realize the automatic fusion and recognition of building remote sensing image. Simulation results show that this method has high precision in constructing remote sensing image fusion and good positioning performance in constructing remote sensing image feature points.

Remote Sensing Image Fusion Using Multi-Scale Convolutional Neural Network

In this paper, a novel remote sensing (RS) image fusion algorithm based on Multi-scale convolutional neural network is proposed. The most important innovation is that the proposed remote sensing image fusion method utilizes a set of convolutional neural networks (CNN) to perform multi-scale image analysis on each band of a multispectral image in order to extract the typical characteristics of different band of multispectral images. In addition, to prevent losing the information of the original image, the max-pooling layer of the traditional CNN is replaced with a standard convolutional layer, and the standard convolutional layer has one step size of 2. The RS image fusion results presented in this paper demonstrate that the proposed method is not only competitive with the most advanced methods, but also superior to other classical methods.

Semi-Supervised Remote Sensing Image Fusion Method Combining Siamese Structure with Generative Adversarial Networks

Semi-Supervised Remote Sensing Image Fusion Method Combining Siamese Structure with Generative Adversarial Networks

A pixel-level fusion method for multi-source optical remote sensing image combining the principal component analysis and curvelet transform

With the availability of multi-sensor, multi-temporal, multi-resolution and multi-spectral images from operational Earth observation satellites, remote sensing image fusion has become a valuable tool. The goal of remote sensing image fusion is to integrate complementary information from multi-source data such that the new images are more suitable for human visual perception and computer-processing tasks such as segmentation, feature extraction, and object recognition. In this paper, a pixel-level remote sensing image fusion method is proposed, which is based on combining the principal component analysis (PCA) and the curvelet transformation (CT). First, the multi-spectral image with low-spatial-resolution is transformed by PCA and principal components are obtained. Second, the panchromatic image with high-spatial-resolution and the principal components of the multi-spectral image are respectively merged with the curvelet transform. Finally, the fused image is obtained by inverse CT and inverse PCA. The experiments using Landsat-8 OLI multi-spectral and panchromatic image show that, compared with the traditional methods such as the WT-based method, the IHS-based method, the HPF-based method, the BT-based method, the PCA-based method and the CT-based method, the results of the proposed method preserve the spatial details while preserving more spectral information of the original image.

Smart city oriented remote sensing image fusion methods based on convolution sampling and spatial transformation

Aiming at the existing problem of losing spatial details and spectral information in remote sensing image fusion, a method of remote sensing image fusion based on convolution sampling transformation is proposed. Firstly, two images to be fused are convoluted, sampled, filtered hierarchically, and decomposed into different sub-images at different levels. Next, those sub-images are fused according to the corresponding locations, and then reconstructed to be a fused image. For images with more complex terrains and objects, the requirements for spatial details and spectral information are higher. So we propose to further transform the fused image and panchromatic image in space. The first component of the former is replaced by that of the latter. Finally, the fused image is obtained after performing inverse transformation. The experimental results show that the fusion effect of two proposed methods is better than that of traditional image fusion algorithms, such as PCA transform, HIS transform, wavelet transform and so on. Compared with the previous method, the latter method has higher resolution but less spectral information in the case of more complex terrains and objects for two proposed methods. They are effective remote sensing image fusion methods.

Remote sensing image fusion method based on discrete wavelet and multiscale morphological transform in the IHS color space

A remote sensing image fusion method is proposed to enhance fusion image quality and fusion efficiency. This method is based on the combination of discrete wavelet and multiscale morphological transform in the IHS color space. First, the three-band multispectral image is converted into the IHS color space to obtain the intensity (I), hue (H), and saturation (S) components. Second, the discrete wavelet transform is used to decompose the I component and PAN images, each of which is decomposed into a set of high-frequency detail images at different scales and one low-frequency approximation image. Then, the dual-channel multiscale morphological transform is constructed based on the idea of a trous algorithm. Upon this, the high-frequency detail images are further decomposed by multiscale morphological gradient transform to extract the edge details information. Two low-frequency approximate images are decomposed by multiscale morphological top-hat and bottom-hat transform to extract the light and dark details information. The decomposition coefficients are fused by designed fusion rules. Finally, the corresponding inverse transform is conducted to reconstruct the fused image. The experimental results have shown that the proposed method performs well, in both subjective and objective evaluations. Compared with other seven hybrid methods popularly applied in image fusion, the proposed method is superior in the 12 indicators. In addition, it has significantly reduced the computational cost in the fusion process. The results have proved that the proposed method not only fully preserves the spectral information of the source MS image but also inherits the rich spatial details of the source PAN image while, achieves high efficiency in image fusion.

A new remote sensing image fusion method combining principal component analysis and curvelet transform

In the framework of multi-scale decomposition, a new multi-source image fusion algorithm based on principal component analysis (PCA) and curvelet transform is proposed for pixel-level image fusion. Firstly the low-resolution multi-spectral image is transformed by PCA and principal components are obtained. Secondly the high-resolution image and the first three principal components of the multi-spectral image are respectively merged with curvelet transform and finally the fused image is obtained by inverse PCA transform. This method gives the results of fusion and is compared with the traditional method of PCA, which proves the effectiveness of the method.

Remote Sensing Image Fusion via Boundary Measured Dual-Channel PCNN in Multi-Scale Morphological Gradient Domain

In this paper, a remote sensing image fusion method based on boundary measured dual-channel pulse-coupled neural network (PCNN) in multi-scale morphological gradient (MSMG) domain is proposed. Firstly, the panchromatic (PAN) image is decomposed into three parts, a small-scale image, a large-scale image, and a base image through a co-occurrence filtering (CoF)-based decomposition model. Secondly, an HSI transform is applied in the multispectral (MS) image to obtained intensity, hue and saturation components. Thirdly, a PCNN fusion strategy modulated by MSMG is used to fuse the base image and the intensity component of the MS image. Then, a fused intensity image is obtained by combining the small-scale image, large-scale image and the fused approximate image. Finally, the final fused image can be reconstructed by an inverse HSI transform. Experiments in four datasets demonstrate that the proposed method obtains the best performance in most cases.

Noisy Remote Sensing Image Fusion Based on JSR

Compressed sensing has shown great potential and power in image representation, especially in image reconstruction by sparse representation. Due to complementary information and unavoidable noise existing in synthetic aperture radar (SAR) and other source images, joint sparse representation (JSR) is developed to separate redundancy and complementary information with different properties in source images and obtain a fused image, where image de-noising is done simultaneously owing to that noise is not sparse and cannot be represented by sparse representation. As a result, one noisy remote sensing image fusion method based on JSR is presented in this paper. After obtaining redundant and complementary sub-images by JSR, an improved fusion rule based on pulse coupled neural network (PCNN) is employed to fuse complementary sparse coefficients together. At the same time, because the types of noise in SAR and other source images are different, they can be treated as the complementary information in source images and suppressed at this step. Finally, a fused image can be reconstructed by adding the redundant and fused complementary sub-images. Quantitative and qualitative experimental results show that the proposed method outperforms most of other fusion methods and it is more robust to noise, having better visual effects and values of objective evaluation metrics.

Remote sensing image fusion method based on adaptive injection model

Remote sensing image fusion method based on adaptive injection model

Remote sensing image fusion algorithm based on mutual-structure for joint filtering using saliency detection

Multimodality image fusion provides more comprehensive information and has an increasingly wide range of uses. For the remote sensing image fusion, traditional multiresolution analysis (MRA)-based methods always have insufficiencies in contrast with spatial details. At the same time, traditional sum of modified Laplace may do blocking artifacts. In order to overcome these deficiencies, we propose a remote sensing image fusion method based on the mutual-structure for joint filtering and saliency detection. Our method uses joint filtering to facilitate the correct extraction of the high and low frequency from source images. The saliency detection method also improves the effect of low-frequency fusion, and the high-frequency sub-bands calculate the extended sum of modified Laplace for better fusion. The method is compared with other five classical fusion methods. The experimental results show that the algorithm effectively preserves the structural information and textural information of the image and improves the sharpness of the fused image. It turns out to have many advantages in subjective and objective evaluation.