Panchromatic Image Research Articles

Generative adversarial approach to urban areas NDVI estimation: A case study of Łódź, Poland

Generative adversarial networks (GAN) opened new possibilities for image processing and analysis. In- painting, dataset augmentation using artificial samples, or increasing spatial resolution of aerial imagery are only a few notable examples of utilising GANs in remote sensing (RS). The normalised difference vegetation index (NDVI) ground-truth labels were prepared by combining RGB and NIR orthophotos. The dataset was then utilised as input for a conditional generative adversarial network (cGAN) to perform an image-to-image translation. The main goal of the neural network was to generate an artificial NDVI image for each processed 256 px × 256 px patch using only in- formation available in the panchromatic input. The network achieved a structural similarity index measure (SSIM) of 0.7569 ± 0.1083, a peak signal-to-noise ratio (PSNR) of 26.6459 ± 3.6577 and a root-mean-square error (RSME) of 0.0504 ± 0.0193 on the test set, which should be considered high. The perceptual evaluation was performed to verify the meth- od’s usability when working with a real-life scenario. The research confirms that the structure and texture of the pan- chromatic aerial RS image contain sufficient information for NDVI estimation for various objects of urban space. Even though these results can highlight areas rich in vegetation and distinguish them from the urban background, there is still room for improvement regarding the accuracy of the estimated values. The research aims to explore the possibility of utilising GAN to enhance panchromatic images (PAN) with information related to vegetation. This opens exciting opportunities for historical RS imagery processing and analysis.

Pansharpening Model of Transferable Remote Sensing Images Based on Feature Fusion and Attention Modules.

The purpose of the panchromatic sharpening of remote sensing images is to generate high-resolution multispectral images through software technology without increasing economic expenditure. The specific method is to fuse the spatial information of a high-resolution panchromatic image and the spectral information of a low-resolution multispectral image. This work proposes a novel model for generating high-quality multispectral images. This model uses the feature domain of the convolution neural network to fuse multispectral and panchromatic images so that the fused images can generate new features so that the final fused features can restore clear images. Because of the unique feature extraction ability of convolution neural networks, we use the core idea of convolution neural networks to extract global features. To extract the complementary features of the input image at a deeper level, we first designed two subnetworks with the same structure but different weights, and then used single-channel attention to optimize the fused features to improve the final fusion performance. We select the public data set widely used in this field to verify the validity of the model. The experimental results on the GaoFen-2 and SPOT6 data sets show that this method has a better effect in fusing multi-spectral and panchromatic images. Compared with the classical and the latest methods in this field, our model fusion obtained panchromatic sharpened images from both quantitative and qualitative analysis has achieved better results. In addition, to verify the transferability and generalization of our proposed model, we directly apply it to multispectral image sharpening, such as hyperspectral image sharpening. Experiments and tests have been carried out on Pavia Center and Botswana public hyperspectral data sets, and the results show that the model has also achieved good performance in hyperspectral data sets.

Multi-Scale and Multi-Stream Fusion Network for Pansharpening

Pansharpening refers to the use of a panchromatic image to improve the spatial resolution of a multi-spectral image while preserving spectral signatures. However, existing pansharpening methods are still unsatisfactory at balancing the trade-off between spatial enhancement and spectral fidelity. In this paper, a multi-scale and multi-stream fusion network (named MMFN) that leverages the multi-scale information of the source images is proposed. The proposed architecture is simple, yet effective, and can fully extract various spatial/spectral features at different levels. A multi-stage reconstruction loss was adopted to recover the pansharpened images in each multi-stream fusion block, which facilitates and stabilizes the training process. The qualitative and quantitative assessment on three real remote sensing datasets (i.e., QuickBird, Pléiades, and WorldView-2) demonstrates that the proposed approach outperforms state-of-the-art methods.

A deep unrolling pansharpening method based on spectral consistency and double spatial priors

ABSTRACT In recent years, deep learning (DL) pansharpening methods have become the most popular solution for acquiring high-resolution multispectral (HRMS) images. However, most DL-based methods design the task as black-box network architecture, ignoring the physical meaning of individual network modules. Such a network structure is difficult to extract sufficient spatial details or lacks interpretability. To address this problem, we propose a neural network pansharpening method with interpretable deep spatial detail injection based on the assumption of spectral consistency and double spatial detail priors in HRMS images. First, we assume that the HRMS image consists of a spectral and spatial component together. Spectral and partial spatial details are obtained by upsampling the multispectral (MS) images. Next, a variational model is constructed for image fusion from the LRMS image and the panchromatic (PAN) image to the HRMS image.The alternating direction multiplier method (ADMM) algorithm is then used to optimally solve the variational model to obtain the spatial detail that needs to be injected into the MS image. Finally, the model optimization solving process is expanded into a corresponding neural network structure using the idea of deep unrolling. Each network module corresponds to the solving step of the iterative algorithm. Our proposed pansharpening method is therefore interpretable in terms of spatial detail injection. In our experiments, we find that the proposed method achieves an optimal balance between spectral and spatial quality, and has a strong generalization capability over different datasets, demonstrating the superiority of the proposed pansharpening method.

DICO: Dingo coot optimization-based ZF net for pansharpening

With the recent advancements in technology, there has been a tremendous growth in the usage of images captured using satellites in various applications, like defense, academics, resource exploration, land-use mapping, and so on. Certain mission-critical applications need images of higher visual quality, but the images captured by the sensors normally suffer from a tradeoff between high spectral and spatial resolutions. Hence, for obtaining images with high visual quality, it is necessary to combine the low resolution multispectral (MS) image with the high resolution panchromatic (PAN) image, and this is accomplished by means of pansharpening. In this paper, an efficient pansharpening technique is devised by using a hybrid optimized deep learning network. Zeiler and Fergus network (ZF Net) is utilized for performing the fusion of the sharpened and upsampled MS image with the PAN image. A novel Dingo coot (DICO) optimization is created for updating the learning parameters and weights of the ZF Net. Moreover, the devised DICO_ZF Net for pansharpening is examined for its effectiveness by considering measures, like Peak Signal To Noise Ratio (PSNR) and Degree of Distortion (DD) and is found to have attained values at 50.177 dB and 0.063 dB.

A spatiotemporal transferable image fusion technique for GeoEye-1 satellite imagery

This study proposed a novel technique to solve the problem of color distortion in the fusion of the GeoEye-1 satellite's panchromatic (PAN) and multispectral (MS) images. This technique suggested reducing the difference in radiometry between the PAN and MS images by using modification coefficients for the MS bands in the definition of the intensity (I) equation, which guarantees using only the overlapped wavelengths with the PAN band. These modification coefficients achieve spatiotemporal transferability for the proposed fusion technique. As the reflectance of vegetation is high in the NIR band and low in the RGB bands, this technique suggested using an additional coefficient for the NIR band in the definition of the I equation, which varies based on the ratio of the agricultural features within the image, to indicate the correct impact of vegetation. This vegetation coefficient provides stability for the proposed fusion technique across all land cover classes. This study used three datasets of GeoEye-1 satellite PAN and MS images in Tanta City, Egypt, with different land cover classes (agricultural, urban, and mixed areas), to evaluate the performance of this technique against five different standard image fusion techniques. In addition, it was validated using six additional datasets from different locations and acquired at different times to test its spatiotemporal transferability. The proposed fusion technique demonstrated spatiotemporal transferability as well as great efficiency in producing fused images of superior spatial and spectral quality for all types of land cover.

A multimodal hyper-fusion transformer for remote sensing image classification

The multispectral (MS) and the panchromatic (PAN) images represent complementary and synergistic spatial spectral information, how to make optimal use of the advantages of them has become a hot research topic. This paper proposes a selectable Transformer and Gist CNN network (STGC-Net). It designs a subspace similar recombination module (SSR-Module) based on non-negative matrix factorization (NMF) and the self-attention mechanism for feature decomposition. This can alleviate the redundant information of multi-modal data and extract their own singular and common features. Considering that the MS and the PAN images exhibit different advantageous properties, a selectable self-attention spectral feature extraction module (S3FE-Module) and a multi-stream Gist spatial feature extraction module (MGSFE-Module) are proposed for the different singular features. The former can refine the Transformer’s input and simultaneously characterize the sequence information between channels for the MS image. The latter introduces the positional relationship between local features while extracting spatial features for the PAN image, thereby improving the accuracy of scene classification. Experimental results indicate that the proposed method performs better than the other methods. The relevant code of this paper is provided at: https://github.com/ru-willow/ST-GC-Net.

INVESTIGATING THE IMPACT OF PAN SHARPENING ON THE ACCURACY OF LAND COVER MAPPING IN LANDSAT OLI IMAGERY

Pan Sharpening is normally applied to sharpen a multispectral image with low resolution by using a panchromatic image with a higher resolution, to generate a high resolution multispectral image. The present study aims at assessing the power of Pan Sharpening on improvement of the accuracy of image classification and land cover mapping in Landsat 8 OLI imagery. In this respect, different Pan Sharpening algorithms including Brovey, Gram-Schmidt, NNDiffuse, and Principal Components were applied to merge the Landsat OLI panchromatic band (15 m) with the Landsat OLI multispectral: visible and infrared bands (30 m), to generate a new multispectral image with a higher spatial resolution (15 m). Subsequently, the support vector machine approach was utilized to classify the original Landsat and resulting Pan Sharpened images to generate land cover maps of the study area. The outcomes were then compared through the generation of confusion matrix and calculation of kappa coefficient and overall accuracy. The results indicated superiority of NNDiffuse algorithm in Pan Sharpening and improvement of classification accuracy in Landsat OLI imagery, with an overall accuracy and kappa coefficient of about 98.66% and 0.98, respectively. Furthermore, the result showed that the Gram-Schmidt and Principal Components algorithms also slightly improved the accuracy of image classification compared to original Landsat image. The study concluded that image Pan Sharpening is useful to improve the accuracy of image classification in Landsat OLI imagery, depending on the Pan Sharpening algorithm used for this purpose.

Effects of topography and historical disturbance on canopy height structure of tropical forests in Menglun, Xishuangbanna, China.

With the combination of airborne Lidar and panchromatic images in 1981 and 2021, we investigated the canopy height structure of tropical forests in Menglun sub-reserve in the Xishuangbanna National Nature Reserve of Yunnan Province, and analyzed its relationship with environmental factors by using multiple regression tree (MRT) method. The results showed that forests in the Menglun sub-reserve could be clustered into seven types based on canopy height structures, with tropical rainforest, monsoon evergreen broad-leaved forest, secondary forest, and flood plain forest as the main types. The potential solar radiation, altitude, terrain profile curvature, slope and the brightness value of imageries in 1981 and 2021 were main factors that drove the classification. The tropical seasonal rainforest dominated by Pometia pinnata occupied the largest area in valley and low-land. The monsoon evergreen broad-leaved forest dominated by Castanopsis echinocarpa mainly distributed in the ridge and disturbed areas. The secondary forests had homogeneous canopy surface, which was significantly different from the primary forests. The activities of swidden agriculture about three decades ago had legacy impacts on the physiognomy of secondary forests.

A method to improve full-resolution remote sensing pansharpening image quality assessment via feature combination

Pansharpening is an important fusion method in remote sensing image processing, aiming at obtaining a multispectral image with high spatial resolution by fusing a multispectral image of low spatial resolution with a panchromatic image of high spatial resolution. A persistent problem with this approach concerns how the quality of the fused images obtained by different methods can be assessed. We address this by constructing a full-resolution quality assessment method of pansharpened remote sensing image based on the combination of low-level features and deep-level features. For the former, different distribution parameters are estimated based on three types of image features prior. On the other hand, deep-level features are designed by employing network. Finally, the quality scores are obtained by combining the two types of features with random forest for regression. The experimental results showed that, compared with the state-of-the-art methods, the FCBM method in this paper can improve the PLCC metric by 16.2%, the SROCC metric by 25%, and the RMSE metric by 42.9%.

Intensity mixture and band-adaptive detail fusion for pansharpening

Pansharpening aims to sharpen a low-resolution multispectral (MS) image through a high-resolution single-channel panchromatic (PAN) image to obtain a high-resolution multi-spectral (HRMS) image. However, low correlation between the PAN and MS images, as well as the inaccurate detail injection for each band of MS image are the key problems causing spectral and spatial distortions in pansharpening. To address these issues, a new pansharpening method based on the intensity mixture and band-adaptive detail fusion is proposed. To obtain a mixed-intensity image (T) that has a high correlation with the MS image and maintain the gradient information of the PAN image, the intensity mixture model is constructed by establishing the intensity and gradient constraints between T and the source images. As it is hard to obtain a proper degradation filter in the model, a filter estimation algorithm is designed by the distribution alignment. To inject the details that match the point spread function of the sensor, a band-adaptive detail fusion algorithm is presented to fuse the details extracted from T with those from the MS image for each band. Furthermore, as there are far fewer details in the MS image than in T, a detail enhancement algorithm is proposed to enhance the details proportionally. The final HRMS image is obtained by injecting the fused details into the upsampled MS image. Extensive experiments show that the proposed method can efficiently achieve the best results in fusion quality compared to state-of-the-art methods. The code is availabe at https://github.com/yotick/IMBD.

Land Cover Classification Improvements by Remote Sensing Data Fusion

Computer processing and analysis of satellite data is an urgent task of the science of remote sensing of the earth. Such processing can range from adjusting the contrast and brightness of the images of an amateur photographer to a group of scientists using neural network classification to determine the types of minerals in a hyperspectral satellite image. This article implements a method of satellite data fusion, which improves the digital image interpretation and image quality for further analysis. For fusion, a multispectral image with a resolution of 30 m Landsat 5 with 6 channels was taken, with three more significant and informative in their composition were used, as well as a panchromatic (monochrome) image with a resolution of 15 m. To evaluate the resolution of the images and the resulting images before and after the image fusion algorithm, image slices along a straight line and intersecting buildings, green mass, roads and industrial areas presented. For testing, test territories taken from Google Earth and the field work results.

Performance evaluation of image fusion techniques and implementation of new fusion technique for remote sensing satellite data

The technique of integrating images from two or more sensors that were taken from the same place or the same object is known as image fusion. The goal is to get more spectral and spatial information from the combined image as a whole than from the individual images. It is required to fuse the images in order to improve the spatial and spectral quality of both panchromatic and multispectral images. This study introduces a novel method for fusing remote sensing images that combines L0 smoothing, NSCT (Non-subsampled Contourlet Transform), SR (Sparse Representation), and MAR (Max absolute rule). The multispectral and panchromatic images are initially divided into lower and higher frequency components using the L0 smoothing filter as the method of fusion. The fusion process is then carried out, utilising a technique that combines NSCT and SR to fuse low-frequency components. Similar to this, the Max-absolute fusion rule is used to fuse high-frequency components. In conclusion, the disintegration of fused low-frequency and high-frequency data yields the final image. Our method yields an enhanced outcome in terms of the correlation coefficient, Entropy, spatial frequency, and fusion of mutual information for both the term of picture quality enhancement and visual evaluation. This suggested approach produces superior outcomes after execution. This study makes use of the Landsat-7ETM+, IKONOS, and Quick Bird datasets. Different satellites are used to take each image. There have been two examples of each image used. In comparison to previous Traditional Methods, the proposed image fusion techniques’ output has a quality that is more than 20% higher.

Panchromatic and Multispectral Image Fusion Combining GIHS, NSST, and PCA

Spatial and spectral information are essential sources of information in remote sensing applications, and the fusion of panchromatic and multispectral images effectively combines the advantages of both. Due to the existence of two main classes of fusion methods—component substitution (CS) and multi-resolution analysis (MRA), which have different advantages—mixed approaches are possible. This paper proposes a fusion algorithm that combines the advantages of generalized intensity–hue–saturation (GIHS) and non-subsampled shearlet transform (NSST) with principal component analysis (PCA) technology to extract more spatial information. Therefore, compared with the traditional algorithms, the algorithm in this paper uses PCA transformation to obtain spatial structure components from PAN and MS, which can effectively inject spatial information while maintaining spectral information with high fidelity. First, PCA is applied to each band of low-resolution multispectral (MS) images and panchromatic (PAN) images to obtain the first principal component and to calculate the intensity of MS. Then, the PAN image is fused with the first principal component using NSST, and the fused image is used to replace the original intensity component. Finally, a fused image is obtained using the GIHS algorithm. Using the urban, plants and water, farmland, and desert images from GeoEye-1, WorldView-4, GaoFen-7 (GF-7), and Gaofen Multi-Mode (GFDM) as experimental data, this fusion method was tested using the evaluation mode with references and the evaluation mode without references and was compared with five other classic fusion algorithms. The results showed that the algorithms in this paper had better fusion performances in both spectral preservation and spatial information incorporation.

SPRINT: Spectra Preserving Radiance Image Fusion Technique using holistic deep edge spatial attention and Minnaert guided Bayesian probabilistic model

Remote Sensing Image Fusion produces a high resolution multispectral image by merging panchromatic image with the corresponding low resolution multispectral counterpart. The limitation of the existing image fusion techniques is that it lacks in maintaining the spectral characteristics of the multispectral image in the fused output. The motivation of our proposed work is to develop balanced and robust image fusion method, named Spectra Preserving Radiance Image Fusion Technique (SPRINT). The technique developed extracts deep edge map from medoid intensity matched image using Holistic Nested Edge Detection (HNED). Minnaert function is applied on denoised panchromatic radiance image along with Digital Elevation Model (DEM) and solar angles’ computation to determine the surface topography. SPRINT’s core design emphasizes on holistic deep edges for spatial attention and terrain guidance using minnaert parameter at each image pixel in a Spectra Preserving Bayesian Probabilistic Model. The unique data pre-processing engine generates fusion ready representative datasets to trigger SPRINT processing workflow. The Indian Cartosat-1 panchromatic and Resourcesat-2/2A multispectral sensors’ datasets along with IKONOS and USGS-NASA’s Landsat-8 OLI images covering diverse landscapes are used for image fusion evaluation and assessment. It has been found that SPRINT’s fusion performance is superior to the state-of-the-art (SOTA) image fusion methods in terms of both visual effects and quantitative metrics. The Normalized Difference Vegetation Index (NDVI) using SPRINT’s fused radiance image tends to have negligible deviation at various classes with respect to reference NDVI. It has been observed that SPRINT derived surface reflectance values have close agreement with original reflectance measurements.

Multi-scale detail injection-based improved generative adversarial networks for pansharpening

ABSTRACT Raw multi-source data include multi-channel multi-spectral images as well as single-channel panchromatic images. Raw multispectral images are rich in spectral detail but low in resolution, whereas raw panchromatic images have high resolution but lack spectral information. Pansharpening is the process of using raw panchromatic and multispectral images to obtain highly accurate remote sensing images. Compared with traditional fusion methods, deep learning-based pansharpening methods perform better. Therefore, this study uses an unsupervised fusion algorithm based on an improved generative adversarial network (GAN) and names it the multi-scale detail injection generative adversarial network (MSDI-GAN). This algorithm uses a network structure with two discriminators and a generator, and the spatial and spectral information of the original multi-source data is retained separately. Each convolutional layer of the generator is used to extract multi-scale feature information of the original data, and injects them into the corresponding discriminator layers of the spatial and spectral discriminator. This method can deepen the exchange of gradient information between the generator and discriminators, optimize the actual network training process and accelerate network convergence. Extensive experiments are conducted to evaluate the performance of our algorithm, and to compare with other algorithms. Based on a qualitative visual effects evaluation of the experimental results and a comparison of quantitative evaluation indicators, it is demonstrated that our method achieves superior results in the pansharpening of remote sensing images.

Panchromatic and multispectral image fusion for remote sensing and earth observation: Concepts, taxonomy, literature review, evaluation methodologies and challenges ahead

Panchromatic and multispectral image fusion, termed pan-sharpening, is to merge the spatial and spectral information of the source images into a fused one, which has a higher spatial and spectral resolution and is more reliable for downstream tasks compared with any of the source images. It has been widely applied to image interpretation and pre-processing of various applications. A large number of methods have been proposed to achieve better fusion results by considering the spatial and spectral relationships among panchromatic and multispectral images. In recent years, the fast development of artificial intelligence (AI) and deep learning (DL) has significantly enhanced the development of pan-sharpening techniques. However, this field lacks a comprehensive overview of recent advances boosted by the rise of AI and DL. This paper provides a comprehensive review of a variety of pan-sharpening methods that adopt four different paradigms, i.e., component substitution, multiresolution analysis, degradation model, and deep neural networks. As an important aspect of pan-sharpening, the evaluation of the fused image is also outlined to present various assessment methods in terms of reduced-resolution and full-resolution quality measurement. Then, we conclude this paper by discussing the existing limitations, difficulties, and challenges of pan-sharpening techniques, datasets, and quality assessment. In addition, the survey summarizes the development trends in these areas, which provide useful methodological practices for researchers and professionals. Finally, the developments in pan-sharpening are summarized in the conclusion part. The aim of the survey is to serve as a referential starting point for newcomers and a common point of agreement around the research directions to be followed in this exciting area.

Pan-sharpening of remote sensing images based on gradient projection and cross fusion

ABSTRACT Pan-sharpening is the fusion of panchromatic (PAN) image and multispectral (MS) image through certain rules to generate high-resolution multispectral (HRMS) image with vivid spatial details and uniform spectral distribution. It has become an important technology in remote sensing image processing. Recently, convolutional neural networks based on deep learning have achieved remarkable results in the field of pan-sharpening. Guided by this method, this paper proposes a pan-sharpening network PSAM-NET for remote sensing images based on depth expansion combined with Cross-Attention Fusion. It consists of gradient projection and the main image fusion module. The gradient projection mainly generates the fusion module by stacking low-resolution multispectral images and panchromatic images alternately, and two deep prior regularized optimization problem formulas are respectively solved by the gradient projection algorithm. The other principal image fusion module realizes double branch fusion, which is mainly composed of cross attention fusion and channel attention fusion, to produce an excellent fusion effect. The simulation and real data experiments were performed on the standard datasets WV2, GF-2, and QB. The qualitative analysis and quantitative comparison with the classical pan-sharpening method proved that the spatial information of the image obtained by this method is complete. The spectral distribution is uniform, and the evaluation index also has some advantages.

F-UNet++: Remote Sensing Image Fusion Based on Multipurpose Adaptive Shuffle Attention and Composite Multi-Input Reconstruction Network

The fusion of multispectral (MS) and panchromatic (PAN) images is of great significance for the construction of high-resolution remote sensing images. Because of differences in sensors, no single MS or PAN image can express the complete information of a scene. Therefore, it is a key issue to fuse MS images containing rich spectral content and PAN images with spatial information to construct a high-resolution MS image. In this work, an adaptive shuffle attention (ASA) module and an optimized UNet++ are combined in a fusion-UNet++ (F-UNet++) framework for the problem of MS and PAN image fusion. This ASA module can focus on important information in the mixed domain and adjust the dimensions of tensors. F-UNet++ includes a multiscale feature extraction module, multiscale feature fusion module, and image reconstruction module. The multiscale feature extraction module obtains spectral and spatial information, the multiscale feature fusion module fuses spectral and spatial information, and a composite multi-input image reconstruction module (CMI-UNet++) reconstructs the final image. By combining the ASA attention module, the loss of feature information can be reduced to enhance the fidelity of the spectral and spatial information of the fused image. Experiments show that F-UNet++ is qualitatively and quantitatively superior to current image fusion methods. (The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Josephing/F-UNet</uri> ).

Pansharpening Based on Adaptive High-Frequency Fusion and Injection Coefficients Optimization

The purpose of pansharpening is to fuse a multispectral (MS) image with a panchromatic (PAN) image to generate a high spatial-resolution multispectral (HRMS) image. However, the traditional pansharpening methods do not adequately take consideration of the information of MS images, resulting in inaccurate detail injection and spectral distortion in the pansharpened results. To solve this problem, a new pansharpening approach based on adaptive high-frequency fusion and injection coefficients optimization is proposed, which can obtain an accurate injected high-frequency component (HFC) and injection coefficients. First, we propose a multi-level sharpening model to enhance the spatial information of the MS image, and then extract the HFCs from the sharpened MS image and PAN image. Next, an adaptive fusion strategy is designed to obtain the accurate injected HFC by calculating the similarity and difference of the extracted HFCs. Regarding the injection coefficients, we propose injection coefficients optimization scheme based on the spatial and spectral relationship between the MS image and PAN image. Finally, the HRMS image is obtained through injecting the fused HFC into the upsampled MS image with the injection coefficients. Experiments with simulated and real data are performed on IKONOS and Pléiades datasets. Both subjective and objective results indicate that our method has better performance than state-of-the-art pansharpening approaches.