Resolution Multispectral Images Research Articles

Unsupervised multi-level spatio-spectral fusion transformer for hyperspectral image super-resolution

Fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) is widely used for HSI super-resolution. Recent works still face problems in exploring global spatio-spectral correlation and lack effective utilization of multi-level features of the inputs (i.e., HR-MSI and LR-HSI), which results in a lack of similarity between the reconstruction and the inputs, ultimately causing significant spatio-spectral distortion. To solve these problems, we design an Unsupervised Multi-level Spatio-spectral Fusion Transformer (UMSFT). In UMSFT, a novel multi-level features fusion strategy is constructed, which fuses the hierarchy features of the inputs via proposed Spatio-spectral Feature Fusion Attention Blocks (S2F-FAB) in a level-by-level manner, thereby fully exploring the interaction between the hierarchical features. The S2F-FAB is specially designed for HSI-MSI fusion consisting of two components: (1) a spatial fusion module (Spa-FM) is designed for spatial domain fusion, and its output is set as Values (V) of a transformer; (2) a novel spectral feature cross attention (Spe- FCA) formulates the features of LR-HSI and HR-MSI as Queries (Q) and Keys (K), respectively, and achieves spectral domain fusion by applying attention mechanism along the spectral dimension. Incorporating spatial detail reconstruction and spectral feature integration into the attention mechanism, the S2F-FAB efficiently exploits the spatio-spectral correlation between target HR-HSI and inputs. Experimental results on three public datasets and our real-world images show the superiority of our method as compared with eleven state-of-the-art methods. Codes will be available at https://github.com/Caoxuheng/HIFtool.

Semi-blind hyperspectral and multispectral image fusion via generalized inverse matrix optimization

Combining a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI) is an economical way to obtain a high spatial resolution hyperspectral image (HR-HSI). However, existing fusion methods often rely on a critical variable, the point spread function (PSF), which is difficult to obtain in practice. Additionally, these methods often require a long fusion time and have many parameters that need re-debugging on different datasets to achieve satisfactory results. Many existing methods lack robustness in handling complex scenarios, such as complex PSF, changing spatial downsampling factors, and large-scale datasets. These issues can result in degraded performance or render the methods inapplicable to practical scenarios. Therefore, we propose an efficient and parameterless semi-blind fusion method based on generalized inverse matrix optimization (GIMO), which is robust for multiple complex situations. Our method uses the definition of generalized inverse matrix and the relationship among HSI, MSI, and HR-HSI to approximate the function module of PSF. Then, we calculate the generalized inverse matrix of the spectral response function (SRF). Finally, we optimize the generalized inverse matrix of SRF using the Sylvester equation and multiply it with MSI to obtain HR-HSI. This is the first proposed method to achieve hyperspectral image super-resolution by optimizing the inverse matrix of the spectral response function. Experiments on multiple simulated datasets and one real dataset have validated that our GIMO method outperforms existing state-of-the-art methods, especially regarding fusion time and robustness for various complex scenarios.

Spectra-phenology integration for high-resolution, accurate, and scalable mapping of foliar functional traits using time-series Sentinel-2 data

Foliar functional traits are essential for understanding plant adaptation strategies and ecosystem function. Due to limited in-situ observational data, there is a growing interest in upscaling these traits from field sites to regional and global levels. However, limitations persist: (1) global/national scale upscaling that relies on plant functional type (PFT) maps, environmental variables or coarse resolution multispectral images, which fail to capture local-scale trait variability; (2) airborne imaging spectroscopy that enables high-resolution and accurate mapping but is restricted to site scale and is costly; and (3) multispectral satellites like Sentinel-2 that offer global coverage but have limited spectral bands and resolution. While previous research has demonstrated the connection between traits and vegetation phenology, our study seeks to build upon this foundation by further exploring the integration of phenological information for large-scale trait prediction. We examined the integration of Sentinel-2 data with its time series (for phenology information) to map 12 foliar functional traits across 14 National Ecological Observatory Network (NEON) sites in the eastern United States. Our results show that time-series Sentinel-2 models effectively capture the variance in these 12 traits (R2 = 0.60–0.80) when compared with benchmark trait data generated by state-of-the-art airborne imaging spectroscopy. The models adequately capture considerable trait variations observed within sites and PFTs. Our approach outperforms existing methods that rely on environmental variables, or a single Sentinel-2 image as predictors across examined NEON sites in eastern United States. Interestingly, including environmental variables in our models does not significantly improve predictive power. Further analysis reveals that a ‘fast-slow’ principal axis predominantly explains the covariation in Enhanced Vegetation Index amplitude (a proxy for leaf longevity), leaf mass per area, and leaf nitrogen content across PFTs. This finding highlights the importance of incorporating phenological information for trait mapping and suggests a potential mechanism underlying these spectra-based models. Our proposed method, which simultaneously achieves high accuracy, large-scale scalability, and high spatial resolution, represents a promising avenue for future global trait mapping. Validation on a larger scale to fully realize its potential in addressing fundamental ecological questions will be a key future focus.

Crop mapping in smallholder farms using unmanned aerial vehicle imagery and geospatial cloud computing infrastructure

Smallholder farms are major contributors to agricultural production, food security, and socio-economic growth in many developing countries. However, they generally lack the resources to fully maximize their potential. Subsequently they require innovative, evidence-based and lower-cost solutions to optimize their productivity. Recently, precision agricultural practices facilitated by unmanned aerial vehicles (UAVs) have gained traction in the agricultural sector and have great potential for smallholder farm applications. Furthermore, advances in geospatial cloud computing have opened new and exciting possibilities in the remote sensing arena. In light of these recent developments, the focus of this study was to explore and demonstrate the utility of using the advanced image processing capabilities of the Google Earth Engine (GEE) geospatial cloud computing platform to process and analyse a very high spatial resolution multispectral UAV image for mapping land use land cover (LULC) within smallholder farms. The results showed that LULC could be mapped at a 0.50 m spatial resolution with an overall accuracy of 91%. Overall, we found GEE to be an extremely useful platform for conducting advanced image analysis on UAV imagery and rapid communication of results. Notwithstanding the limitations of the study, the findings presented herein are quite promising and clearly demonstrate how modern agricultural practices can be implemented to facilitate improved agricultural management in smallholder farmers.

Triple disentangled network with dual attention for remote sensing image fusion

Remote sensing applications, such as detection and recognition of objects, need a large number of high spatial resolution (HR) images. As a feasible technique, image fusion is considered to produce HR images. However, most of remote sensing image fusion methods based on deep neural network (DNN) are mainly limited by the extraction of spatial and spectral features. These methods ignore the redundancy among these features, which causes spatial and spectral distortions in the fused image. In this paper, we propose a novel image fusion method based on a triple disentangled network (TDNet) with dual attention to reduce the redundancy in the extracted features. In the proposed method, it is assumed that the information in panchromatic (PAN) and low spatial resolution multispectral (LRMS) images can be encoded as the spatial, spectral, and common features. Specifically, we construct a triple-stream network to extract these features. To efficiently model the spatial and spectral information in PAN and LRMS images, local–global attention and interdependency attention are designed and integrated into the network. Then, the redundancy among these features is reduced by disentangled learning, in which these features are recombined to reconstruct the PAN and LRMS images. Besides, these features should complement each other. So, we utilize the maximal coding rate reduction to balance the redundancy and complementarity among them. Finally, all features are recombined to synthesize the high spatial resolution multispectral image. The experimental results demonstrate that the proposed TDNet has a superior performance in terms of qualitative and quantitative evaluations. The code link is https://github.com/RSMagneto/TDNet.

Maize Evapotranspiration Estimates Using Planet Dove Mini-Satellites and Field-Level InfraRed Thermometers

Highlights Multispectral images from mini-satellites can be used in a reflectance-based crop coefficient model to estimate crop water use. Surface reflectance from mini-satellites and temperature from in-situ infrared thermometers can be used in a land surface energy balance model to more accurately estimate maize actual evapotranspiration. It is feasible to monitor daily maize actual evapotranspiration using Planet Dove mini-satellites on cloud-free days. Abstract. Water scarcity, climate change, and the need to sustain irrigated agriculture demand improved methods to manage irrigation water. Irrigation scheduling can be improved when remote sensing (RS) images are used because RS captures the actual crop field conditions. Mini-satellites (mS) are small spaceborne RS platforms that can provide high spatial and temporal resolution multispectral (MS) images. These multispectral images can be used to estimate maize actual evapotranspiration (ETa) on a daily basis. Thus, the main objective of this study was to assess the accuracy of maize ETa estimates when mS MS images were used in two different ETa algorithms. This study was conducted at two research sites in northern Colorado, USA. A reflectance-based crop coefficient and a one-source surface energy balance (OSSEB) models were applied, using MS Planet Dove images, to map maize ETa. An Eddy Covariance energy balance system was used to assess the accuracy of distributed ETa estimates. Results indicate that the reflectance-based crop coefficient method can be used with mS MS images yielding acceptable estimates. Both ETa models performed better in the case of the sub-surface irrigated maize. Further, OSSEB-based estimates of ETa model were more accurate over the surface irrigated maize and homogeneous soils; when surface reflectance images were paired with radiometric surface temperature data from ground-based infrared thermometers. These results imply that potentially maize ETa could be mapped, on a daily basis with good accuracy, using Planet Dove mS-based MS reflectance images and ground-based radiometric surface temperatures. Keywords: Agricultural water management, Crop water use, Irrigation, Land surface energy balance, Reflectance-based crop coefficients, Remote sensing.

CS2DIPs: Unsupervised HSI Super-Resolution Using Coupled Spatial and Spectral DIPs.

In recent years, fusing high spatial resolution multispectral images (HR-MSIs) and low spatial resolution hyperspectral images (LR-HSIs) has become a widely used approach for hyperspectral image super-resolution (HSI-SR). Various unsupervised HSI-SR methods based on deep image prior (DIP) have gained wide popularity thanks to no pre-training requirement. However, DIP-based methods often demonstrate mediocre performance in extracting latent information from the data. To resolve this performance deficiency, we propose a coupled spatial and spectral deep image priors (CS2DIPs) method for the fusion of an HR-MSI and an LR-HSI into an HR-HSI. Specifically, we integrate the nonnegative matrix-vector tensor factorization (NMVTF) into the DIP framework to jointly learn the abundance tensor and spectral feature matrix. The two coupled DIPs are designed to capture essential spatial and spectral features in parallel from the observed HR-MSI and LR-HSI, respectively, which are then used to guide the generation of the abundance tensor and spectral signature matrix for the fusion of the HSI-SR by mode-3 tensor product, meanwhile taking some inherent physical constraints into account. Free from any training data, the proposed CS2DIPs can effectively capture rich spatial and spectral information. As a result, it exhibits much superior performance and convergence speed over most existing DIP-based methods. Extensive experiments are provided to demonstrate its state-of-the-art overall performance including comparison with benchmark peer methods.

A Triple-Double Convolutional Neural Network for Panchromatic Sharpening.

Pansharpening refers to the fusion of a panchromatic (PAN) image with a high spatial resolution and a multispectral (MS) image with a low spatial resolution, aiming to obtain a high spatial resolution MS (HRMS) image. In this article, we propose a novel deep neural network architecture with level-domain-based loss function for pansharpening by taking into account the following double-type structures, i.e., double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the PAN image can be fully exploited and utilized to progressively inject into the low spatial resolution MS (LRMS) image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach. The code is available at https://github.com/liangjiandeng/TDNet.

ZeRGAN: Zero-Reference GAN for Fusion of Multispectral and Panchromatic Images.

In this article, we present a new pansharpening method, a zero-reference generative adversarial network (ZeRGAN), which fuses low spatial resolution multispectral (LR MS) and high spatial resolution panchromatic (PAN) images. In the proposed method, zero-reference indicates that it does not require paired reduced-scale images or unpaired full-scale images for training. To obtain accurate fusion results, we establish an adversarial game between a set of multiscale generators and their corresponding discriminators. Through multiscale generators, the fused high spatial resolution MS (HR MS) images are progressively produced from LR MS and PAN images, while the discriminators aim to distinguish the differences of spatial information between the HR MS images and the PAN images. In other words, the HR MS images are generated from LR MS and PAN images after the optimization of ZeRGAN. Furthermore, we construct a nonreference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss. Through the minimization of the total loss, the spatial details in the HR MS images can be enhanced efficiently. Extensive experiments are implemented on datasets acquired by different satellites. The results demonstrate that the effectiveness of the proposed method compared with the state-of-the-art methods. The source code is publicly available at https://github.com/RSMagneto/ZeRGAN.

Crop height estimation of sorghum from high resolution multispectral images using the structure from motion (SfM) algorithm

Crop height estimation of sorghum from high resolution multispectral images using the structure from motion (SfM) algorithm

A Swin Transformer with Dynamic High-Pass Preservation for Remote Sensing Image Pansharpening

Pansharpening is a technique used in remote sensing to combine high-resolution panchromatic (PAN) images with lower resolution multispectral (MS) images to generate high-resolution multispectral images while preserving spectral characteristics. Recently, convolutional neural networks (CNNs) have been the mainstream in pansharpening by extracting the deep features of PAN and MS images and fusing these abstract features to reconstruct high-resolution details. However, they are limited by the short-range contextual dependencies of convolution operations. Although transformer models can alleviate this problem, they still suffer from weak capability in reconstructing high-resolution detailed information from global representations. To this end, a novel Swin-transformer-based pansharpening model named SwinPAN is proposed. Specifically, a detail reconstruction network (DRNet) is developed in an image difference and residual learning framework to reconstruct the high-resolution detailed information from the original images. DRNet is developed based on the Swin Transformer with a dynamic high-pass preservation module with adaptive convolution kernels. The experimental results on three remote sensing datasets with different sensors demonstrate that the proposed approach performs better than state-of-the-art networks through qualitative and quantitative analysis. Specifically, the generated pansharpening results contain finer spatial details and richer spectral information than other methods.

Extracting hydrothermally altered information using WorldView-3 data: a case study of Huitongshan, NW Gansu, China

Introduction: The Huitongshan skarn-type deposits, in which ore bodies primarily occur in the outer contact zone between K-feldspar granite and marble in the Beishan area, are evidently related to hydrothermal alteration of the surrounding rock. Key mineral alteration processes include serpentinization, epidotization, chloritization, carbonatization, jarosite, ferritization, and hematitization.Methods: WorldView-3 (WV-3), a satellite-recorded high-spatial resolution multispectral image, has been widely used in the exploration and prediction of different types of deposits around the world. In this study, WV-3 multispectral images were used to extract the spatial distribution data of the main altered minerals in the Huitongshan area. Dedicated radiometric calibration, atmospheric correction, and image fusion were used to pre-process the extracted spectral information related to hydrothermal alteration. In addition, directed principal component analysis (PCA) and a unique mineral index were designed based on the effective use of the WV-3 data band corresponding to the spectral absorption characteristics of altered minerals.Results: The findings of this study show that the PCA model and mineral index pro-posed herein are reliable both in theory and for practically obtaining extraction information. Additionally, the WV-3 data are well suited for identifying hydroxy-bearing alterations with rich short-wave infrared bands that distinguish Fe-OH–bearing alterations from Mg-OH–bearing alterations. The results obtained were applied to identify potential targets for skarn-type copper deposits and the implementation of prospecting practices.Discussion: This study provides a basis for the application of WV-3 data as an important and effective tool for alteration information extraction and determination of prospecting practice, thereby proving the validity of multispectral remote sensing images in mineral resource exploration.

Multispectral and hyperspectral image fusion based on low-rank unfolding network

Recently, deep unfolding networks (DUNs) have been applied to the fusion of low spatial resolution hyperspectral (LR HS) and high spatial resolution multispectral (HR MS) images and achieved satisfactory high spatial resolution hyperspectral (HR HS) images. However, the low-rank and sparse priors in these networks are not exploited sufficiently. In this paper, we establish an LR HS and HR MS image fusion model based on robust principal component analysis (RPCA), which simultaneously captures the low-rank and sparse properties in HS images. Then, the fusion model is optimized with the alternating direction method of multipliers (ADMM). To make full use of the representation capacity of DUNs, we unfold the derived ADMM algorithm as a network, named low-rank unfolding network (LRU-Net). Specifically, each iteration in ADMM is unfolded as one stage in LRU-Net, in which the low-rank and sparse priors are learned by singular value thresholding (SVT) and sparse module, respectively. Finally, all features from all stages are integrated to produce the desired HR HS image. Three benchmark datasets were chosen for comparison to demonstrate the effectiveness of the proposed LRU-Net. The experimental results demonstrate that LRU-Net performs better in terms of both qualitative and quantitative results compared to state-of-the-art fusion methods. The source code is publicly available at https://github.com/RSMagneto/LRU-Net.

Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion for remote sensing image

ABSTRACT Remote sensing image (RSI) with concurrently high spatial, temporal, and spectral resolutions cannot be produced by a single sensor. Multisource RSI fusion is a convenient technique to realize high spatial resolution multispectral (MS) images (spatial spectral fusion, i.e. SSF) and high temporal and spatial resolution MS images (spatiotemporal fusion, i.e. STF). Currently, deep learning-based fusion models can only implement SSF or STF, lacking models that perform both SSF and STF. Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion (BAPGF) for RSI are proposed to implement both SSF and STF, namely BPF-MGAN. A bidirectional adaptive-stage feature extraction architecture in fine-scale-to-coarse-scale and coarse-scale-to-fine-scale modes is introduced. The designed BAPGF introduces a previous fusion result-oriented cross-stage-level dual-residual attention fusion strategy to enhance critical information and suppress superfluous information. Adaptive resolution U-shaped discriminators are implemented to feed multiresolution context into the generator. A generalized multitask loss function unlimited by no-reference images is developed to strengthen the model via constraints on the multiscale feature, structural, and content similarities. The BPF-MGAN model is validated on SSF datasets and STF datasets. Compared with the state-of-the-art SSF and STF models, results demonstrate the superior performance of the proposed BPF-MGAN model in both subjective and objective evaluations.

A image fusion and U-Net approach to improving crop planting structure multi-category classification in irrigated area

Some subtle features of planting structures in irrigation areas could only be visible on high-resolution panchromatic spectral images. However, low spatial resolution multispectral image makes it hard to recognize them. It is challenging to accurately obtain crop planting structure when using traditional methods. This paper proposes an extraction method of crop planting structure based on image fusion and U-Net depth semantic segmentation network, which can automatically and accurately extract multi-category crop planting structure information. This method takes Landsat8 commercial multispectral satellite data set as an example, chooses RGB pseudo-color synthetic image which highlights vegetation characteristics, and uses HLS(Hue, Luminance, Saturation), NND(Nearest-Neighbor Diffusion) and G-S(Gram-Schmidt) methods to fuse panchromatic band to obtain 15m high-resolution fusion image to obtain training set and test set, six types of land features including cities and rivers were labeled by manual to obtain the verification set. The training and validation sets are cut and enhanced to train the U-Net semantic segmentation network. Taking the Xiaokaihe irrigation area in Binzhou City, Shandong Province, China, as an example, the planting structure was classified, and the overall accuracy was 87.7%, 91.2%, and 91.3%, respectively. The accuracy of crop planting structures (wheat, cotton, woodland) was 74.2%, 82.5%, 82.3%, and the Kappa coefficient was 0.832, 0.880, and 0.881, respectively. The results showed that the NND-UNet method was suitable for large-scale continuous crop types (wheat, cotton), and the GS-UNet method had a better classification effect in discrete areas of cash crops (Jujube and many kinds of fruit trees).

Density vs. Cover: Which is the better choice as the proxy for plant community species diversity estimated by spectral indexes?

Recent studies have shown the crucial role of plant cover in the spectral signals of remote sensing images. However, how plant species cover affects the spectral-species diversity relationship remains largely unknown. To clarify this, we conducted a study in wetlands and obtained high spatial resolution multispectral images collected by unmanned aerial vehicles (UAVs). Quadrat information was obtained through a field survey. The traditional species diversity indexes and Hill numbers of each quadrat were computed by species density and cover. The mean, standard deviation, and coefficient of variation of the normalized difference vegetation index (NDVI) were computed based on the multispectral images. We employed univariate and multivariate regression models to assess the ability of spectral information to predict species diversity and the explanatory degree of variables to the multivariate regression models were decomposed. The results showed that rather than species density, spectral indexes were more strongly related to species diversity indexes computed by species cover, and the robustness and universality of the methods that predict plant species diversity based on spectral indexes have been improved by considering species cover. Meanwhile, in light of the flexibility of Hill numbers that assign different weights to dominant and rare species in the community, we believe that Hill numbers are the better choice in the spectral-species diversity studies, which can suitable to various communities with different density-cover relationships. In addition, our study also provided an insightful perspective for future spectral-plant species diversity research.

Assessment of vineyard vigour and yield spatio-temporal variability based on UAV high resolution multispectral images

Accurate, timely assessment of the vineyard on a field scale is essential for successful grape yield and quality. Remote sensing can be an effective and useful monitoring tool, as data from sensors on board Unmanned Aerial Vehicles (UAV) can measure vegetative and reproductive growth and thus directly or indirectly detect variability. Through the images obtained from UAV, the Vegetation Indices (VIs) can be calculated and compared with various agronomic characteristics of the vineyard. The objective of this study was to evaluate the multispectral response of the vineyard in three specific phenological phases and to analyse the spatial distribution of vegetative vigour. A multirotor UAV equipped with a camera featuring multispectral sensors was used. Four VIs namely Normalised Difference Vegetation Index (NDVI), Normalised Difference Red Edge (NDRE), Green Normalised Difference Vegetation Index (GNDVI), Modified Soil Adjusted Vegetation Index (MSAVI), were calculated using the georeferenced orthomosaic UAV images. Computer vision techniques were used to segment these orthoimages to extract only the vegetation canopy pixels. High level of agronomic variability within the vineyard was identified. Pearson's coefficient showed a significant correlation between NDVI and NDRE indices and yield since early phenological stages (r = 0.80 and 0.72 respectively), GNDVI at grape ripening (r = 0.83). Shoot pruning weight (SPW) shows the highest values of correlation (r = 0.84) with NDVI during the phenological stage of berries pea size. Simple linear regression techniques were evaluated using VIs as predictors of the SPW, and accurate predictive results were obtained for NDVI and NDRE with RMSE values of 0.18 and 0.24, respectively. Geostatistical analysis was applied to model the spatial variability of SPW, and thus vineyard vigour. Assessing spatial variability and appreciating the level of vigour enables improved vineyard management by increasing sustainability and production efficiency.

Modified Dynamic Routing Convolutional Neural Network for Pan-Sharpening

Based on deep learning, various pan-sharpening models have achieved excellent results. However, most of them adopt simple addition or concatenation operations to merge the information of low spatial resolution multi-spectral (LRMS) images and panchromatic (PAN) images, which may cause a loss of detailed information. To tackle this issue, inspired by capsule networks, we propose a plug-and-play layer named modified dynamic routing layer (MDRL), which modifies the information transmission mode of capsules to effectively fuse LRMS images and PAN images. Concretely, the lower-level capsules are generated by applying transform operation to the features of LRMS images and PAN images, which preserve the spatial location information. Then, the dynamic routing algorithm is modified to adaptively select the lower-level capsules to generate the higher-level capsule features to represent the fusion of LRMS images and PAN images, which can effectively avoid the loss of detailed information. In addition, the previous addition and concatenation operations are illustrated as special cases of our MDRL. Based on MIPSM with addition operations and DRPNN with concatenation operations, two modified dynamic routing models named MDR–MIPSM and MDR–DRPNN are further proposed for pan-sharpening. Extensive experimental results demonstrate that the proposed method can achieve remarkable spectral and spatial quality.

RaGMAN: A relativistic average generative multi-adversarial network for pansharpening

Pansharpening aims to reconstruct a high-resolution multi-spectral image (HRMS) from a low spatial resolution multi-spectral (MS) image and a high spatial resolution single band panchromatic (PAN) image acquired from the same satellite. To solve the spatial information and spectral information of the fused image that are not evenly preserved, this paper proposes a relativistic average generative multi-adversarial network (RaGMAN) for pansharpening. The RaGMAN consists of two parts. One builds a two-stream generator network to generate HRMS images by extracting features from PAN and MS images. The other is a dual discriminator to preserve the spectral and spatial information of the input when performing fusion. To make the generated image contain more spatial information, we propose multiple residual dense block in the generator. At the same time, in order to improve the overall quality of the fused images, two relativistic average discriminators are used in the network. Furthermore, a novel hybrid loss function is introduced to optimize training. Compared with seven state-of-the-art methods on GF-2, QB and WV-3 datasets, experimental results show that the proposed RaGMAN method can produce excellent pansharpening performance in terms of spatial and spectral fidelity.

DICO: Dingo coot optimization-based ZF net for pansharpening

With the recent advancements in technology, there has been a tremendous growth in the usage of images captured using satellites in various applications, like defense, academics, resource exploration, land-use mapping, and so on. Certain mission-critical applications need images of higher visual quality, but the images captured by the sensors normally suffer from a tradeoff between high spectral and spatial resolutions. Hence, for obtaining images with high visual quality, it is necessary to combine the low resolution multispectral (MS) image with the high resolution panchromatic (PAN) image, and this is accomplished by means of pansharpening. In this paper, an efficient pansharpening technique is devised by using a hybrid optimized deep learning network. Zeiler and Fergus network (ZF Net) is utilized for performing the fusion of the sharpened and upsampled MS image with the PAN image. A novel Dingo coot (DICO) optimization is created for updating the learning parameters and weights of the ZF Net. Moreover, the devised DICO_ZF Net for pansharpening is examined for its effectiveness by considering measures, like Peak Signal To Noise Ratio (PSNR) and Degree of Distortion (DD) and is found to have attained values at 50.177 dB and 0.063 dB.