Fusion Of Optical Images Research Articles

Super-resolution water body mapping with a feature collaborative CNN model by fusing Sentinel-1 and Sentinel-2 images

Mapping water bodies from remotely sensed imagery is crucial for understanding hydrological and biogeochemical processes. The identification of water extent is mainly dependent on optical and synthetic aperture radar (SAR) images. However, the use of remote sensing for water body mapping is often undermined by the mixed pixel dilemma inherent to traditional hard classification approaches. At the same time, the presence of clouds in optical imagery and speckle noise in SAR imagery, coupled with the difficulty in differentiating between water-like surfaces and actual water bodies, significantly compromise the accuracy of water body identification. This paper proposes a DEEP feature collaborative convolutional neural network (CNN) for Water Super-Resolution Mapping based on Optical and SAR images (DeepOSWSRM), which collaboratively leverages Sentinel-1 and Sentinel-2 imagery to address the challenges of missing data and mixed pixels. The Sentinel-1 image provides complementary water distribution information for the cloudy areas of the Sentinel-2 image, while the Sentinel-2 image enhances the perception capabilities for small water bodies in the Sentinel-1 image. Using PlanetScope imagery as the true reference data, the effectiveness of the proposed method was assessed through two experimental scenarios: one utilizing synthetic coarse-resolution imagery degraded from Sentinel-1 and Sentinel-2 data and another using actual Sentinel-1 and Sentinel-2 data, encompassing both simulated and real cloud conditions. A comparative analysis was conducted against three state-of-the-art CNN-based water mapping methods and two CNN SRM methods. The findings demonstrate that the proposed DeepOSWSRM method successfully produces accurate, fine-resolution water body maps, with its performance mainly benefiting from the fusion of SAR and optical images.

BCLNet: Boundary contrastive learning with gated attention feature fusion and multi-branch spatial-channel reconstruction for land use classification

The fusion of optical and synthetic aperture radar (SAR) images is not only a crucial method for enhancing land use classification tasks but also forms a fundamental basis for the interpretation of multimodal remote sensing imagery. However, the visual differences caused by the distinct nonlinear radiometric properties of optical and SAR images lead to challenges in effectively integrating multimodal features using current land use classification methods, resulting in lower classification accuracy. On one hand, these approaches may be affected by the heterogeneity among different source images, leading to an inability to fully utilize the semantic content of the fused features. On the other hand, the semantic embedding space does not consider the semantic relationships between different pixels from a global perspective (i.e., across the entire dataset). To address the aforementioned challenges, we propose the boundary contrastive learning classification network (BCLNet). In this framework, we introduce the gate attention fusion (GAF) module, which selectively considers distinctiveness and consistency features in optical and SAR imagery, facilitating effective modal representation fusion. Furthermore, we introduce the multi-branch spatial channel reconstruction (MSCR) module for enhanced feature augmentation, operating in both spatial and channel dimensions with a dual-dimensional feature reconstruction selection mechanism for the fused features. Furthermore, we adopt boundary contrast learning to address the global pixel semantic space embedding problem. Our method is evaluated on the WHU-OPT-SAR dataset and a multi-class scene classification dataset (MSD) that we constructed. Compared to state-of-the-art methods, our approach demonstrates improvements in pixel classification overall accuracy (OA) and mean intersection over union (mIOU).

Spatial and Spectral Analysis of Resampling Algorithms in Image Fusion of Optical and Microwave Satellite Images: A Case Study Over Western Himalayas

Spatial and Spectral Analysis of Resampling Algorithms in Image Fusion of Optical and Microwave Satellite Images: A Case Study Over Western Himalayas

Recognition of small water bodies under complex terrain based on SAR and optical image fusion algorithm

Understanding the spatial and temporal distribution of small water bodies is essential for managing water resources, crafting conservation policies, and preserving watershed ecosystems and biodiversity. However, existing studies often rely on a single remote sensing data source (optical or microwave), focusing on large-scale, flat areas and lacking comprehensive monitoring of small water bodies in complex terrain. Therefore, considering the complementary advantages of multisource remote sensing (multispectral and SAR), this paper proposes a multispectral and SAR fusion algorithm, named Multispectral and SAR Fusion algorithm (MASF), to better capture the remote sensing characteristics of small water bodies in complex areas. Based on this, a dataset containing spectral, texture, and geometric features is constructed, and multi-scale segmentation and random forest algorithms are applied for identification of small water bodies in complex terrain. The results demonstrate that the proposed fusion algorithm MASF exhibits minimal spectral distortion (SAM < 3.5, ERGAS <21, RMSE <0.01) and robust spatial feature enhancement (PSNR >40, SSIM >0.999, CC > 0.99). The Overall Accuracy (OA) and Kappa coefficients for both experimental areas surpassed 0.9. For rivers and reservoirs, both Producer's Accuracy (PA) and User's Accuracy (UA) exceeded 0.9. The UA for agricultural ponds exceeded 0.8. Comparative analysis with three other types of water-related data products shows that the freshwater identification results in this study have certain advantages in local small water bodies. Our research holds significant implications for the utilization of water resources in mountainous areas, prevention and control of floods and floods, as well as the development of aquaculture industry.

Fusion of full-field optical angiography images via gradient feature detection

Full-field optical angiography (FFOA)—a real-time non-invasive imaging technique for extracting biological blood microcirculation information—contributes to an in-depth understanding of the functional and pathological changes of biological tissues. However, owing to the limitation of the depth-of-field (DOF) of optical lenses, existing FFOA imaging methods cannot capture an image containing every blood-flow information. To address this problem, this study develops a long-DOF full-field optical angiography imaging system and proposes a novel multi-focus image fusion scheme to expand the DOF. First, FFOA images with different focal lengths are acquired by the absorption intensity fluctuation modulation effect. Second, an image fusion scheme based on gradient feature detection in a nonsubsampled contourlet transform domain is developed to capture focus features from FFOA images and synthesize an all-focused image. Specifically, FFOA images are decomposed by NSCT into coefficients and low-frequency difference images; thereafter, two gradient feature detection-based fusion rules are used to select the pre-fused coefficients. The experimental results of both phantom and animal cases show that the proposed fusion method can effectively extend the DOF and address practical FFOA image defocusing problems. The fused FFOA image can provide a more comprehensive description of blood information than a single FFOA image.

A Comprehensive Assessment of the Pansharpening of the Nighttime Light Imagery of the Glimmer Imager of the Sustainable Development Science Satellite 1

The Sustainable Development Science Satellite 1 (SDGSAT-1) satellite, launched in November 2021, is dedicated to providing data detailing the “traces of human activities” for the implementation of the United Union’s 2030 Agenda for Sustainable Development and global scientific research. The glimmer imager (GI) that is equipped on SDGSAT-1 can provide nighttime light (NL) data with a 10 m panchromatic (PAN) band and red, green, and blue (RGB) bands of 40 m resolution, which can be used for a wide range of applications, such as in urban expansion, population studies of cities, and economics of cities, as well as nighttime aerosol thickness monitoring. The 10 m PAN band can be fused with the 40 m RGB bands to obtain a 10 m RGB NL image, which can be used to identify the intensity and type of night lights and the spatial distribution of road networks and to improve the monitoring accuracy of sustainable development goal (SDG) indicators related to city developments. Existing remote sensing image fusion algorithms are mainly developed for daytime optical remote sensing images. Compared with daytime optical remote sensing images, NL images are characterized by a large amount of dark (low-value) pixels and high background noises. To investigate whether daytime optical image fusion algorithms are suitable for the fusion of GI NL images and which image fusion algorithms are the best choice for GI images, this study conducted a comprehensive evaluation of thirteen state-of-the-art pansharpening algorithms in terms of quantitative indicators and visual inspection using four GI NL datasets. The results showed that PanNet, GLP_HPM, GSA, and HR outperformed the other methods and provided stable performances among the four datasets. Specifically, PanNet offered UIQI values ranging from 0.907 to 0.952 for the four datasets, whereas GSA, HR, and GLP_HPM provided UIQI values ranging from 0.770 to 0.856. The three methods based on convolutional neural networks achieved more robust and better visual effects than the methods using multiresolution analysis at the original scale. According to the experimental results, PanNet shows great potential in the fusion of SDGSAT-1 GI imagery due to its robust performance and relatively short training time. The quality metrics generated at the degraded scale were highly consistent with visual inspection, but those used at the original scale were inconsistent with visual inspection.

Optical and SAR Image Fusion Based on Complementary Feature Decomposition and Visual Saliency Features

Optical and SAR Image Fusion Based on Complementary Feature Decomposition and Visual Saliency Features

Optimal Fusion of Multispectral Optical and SAR Images for Flood Inundation Mapping through Explainable Deep Learning

In the face of increasing flood risks intensified by climate change, accurate flood inundation mapping is pivotal for effective disaster management. This study introduces a novel explainable deep learning architecture designed to generate precise flood inundation maps from diverse satellite data sources. A comprehensive evaluation of the proposed model is conducted, comparing it with state-of-the-art models across various fusion configurations of Multispectral Optical and Synthetic Aperture Radar (SAR) images. The proposed model consistently outperforms other models across both Sentinel-1 and Sentinel-2 images, achieving an Intersection Over Union (IOU) of 0.5862 and 0.7031, respectively. Furthermore, analysis of the different fusion combinations reveals that the use of Sentinel-1 in combination with RGB, NIR, and SWIR achieves the highest IOU of 0.7053 and that the inclusion of the SWIR band has the greatest positive impact on the results. Gradient-weighted class activation mapping is employed to provide insights into its decision-making processes, enhancing transparency and interpretability. This research contributes significantly to the field of flood inundation mapping, offering an efficient model suitable for diverse applications. This study not only advances flood inundation mapping but also provides a valuable tool for improved understanding of deep learning decision-making in this area, ultimately contributing to improved disaster management strategies.

OPTICAL AND SAR IMAGE FUSION BASED ON VISUAL SALIENCY FEATURES

Abstract. With the expansion of optical and SAR image fusion application scenarios, it is necessary to integrate their information in land classification, feature recognition, and target tracking. Current methods focus excessively on integrating multimodal feature information to enhance the information richness of the fused images, while neglecting the highly corrupted visual perception of the fused results by modal differences and SAR speckle noise. To address this problem, in this paper we propose a novel optical and SAR image fusion framework named Visual Saliency Features Fusion (VSFF). We improved the decomposition algorithm of complementary feature to reduce most of the speckle noise in the initial features, and divide the image into main structure features and detail texture features. For the fusion of main structure features, we reconstruct a visual saliency features map that contains significant information from optical and SAR images, and input it together with the optical image into a total variation constraint model to compute the fusion result and achieve the optimal information transfer. Meanwhile, we construct a new feature descriptor based on Gabor wavelet, which separates meaningful detail texture features from residual noise and selectively preserves features that can improve the interpretability of fusion result. Further a fast IHS transform fusion is used to supplement the fused image with realistic color information. In a comparative analysis with five state-of-the-art fusion algorithms, VSFF achieved better results in qualitative and quantitative evaluations, and our fused images have a clear and appropriate visual perception.

CycleGAN-Based SAR-Optical Image Fusion for Target Recognition

The efficiency and accuracy of target recognition in synthetic aperture radar (SAR) imagery have seen significant progress lately, stemming from the encouraging advancements of automatic target recognition (ATR) technology based on deep learning. However, the development of a deep learning-based SAR ATR algorithm still faces two critical challenges: the difficulty of feature extraction caused by the unique nature of SAR imagery and the scarcity of datasets caused by the high acquisition cost. Due to its desirable image nature and extremely low acquisition cost, the simulated optical target imagery obtained through computer simulation is considered a valuable complement to SAR imagery. In this study, a CycleGAN-based SAR and simulated optical image fusion network (SOIF-CycleGAN) is designed and demonstrated to mitigate the adverse effects of both challenges simultaneously through SAR-optical image bidirectional translation. SAR-to-optical (S2O) image translation produces artificial optical images that are high-quality and rich in details, which are used as supplementary information for SAR images to assist ATR. Conversely, optical-to-SAR (O2S) image translation generates pattern-rich artificial SAR images and provides additional training data for SAR ATR algorithms. Meanwhile, a new dataset of SAR-optical image pairs containing eight different types of aircraft has been created for training and testing SOIF-CycleGAN. By combining image-quality assessment (IQA) methods and human vision, the evaluation verified that the proposed network possesses exceptional bidirectional translation capability. Finally, the results of the S2O and O2S image translations are simultaneously integrated into a SAR ATR network, resulting in an overall accuracy improvement of 6.33%. This demonstrates the effectiveness of SAR-optical image fusion in enhancing the performance of SAR ATR.

Comparative Analysis of Pixel-Level Fusion Algorithms and a New High-Resolution Dataset for SAR and Optical Image Fusion

Synthetic aperture radar (SAR) and optical images often present different geometric structures and texture features for the same ground object. Through the fusion of SAR and optical images, it can effectively integrate their complementary information, thus better meeting the requirements of remote sensing applications, such as target recognition, classification, and change detection, so as to realize the collaborative utilization of multi-modal images. In order to select appropriate methods to achieve high-quality fusion of SAR and optical images, this paper conducts a systematic review of current pixel-level fusion algorithms for SAR and optical image fusion. Subsequently, eleven representative fusion methods, including component substitution methods (CS), multiscale decomposition methods (MSD), and model-based methods, are chosen for a comparative analysis. In the experiment, we produce a high-resolution SAR and optical image fusion dataset (named YYX-OPT-SAR) covering three different types of scenes, including urban, suburban, and mountain. This dataset and a publicly available medium-resolution dataset are used to evaluate these fusion methods based on three different kinds of evaluation criteria: visual evaluation, objective image quality metrics, and classification accuracy. In terms of the evaluation using image quality metrics, the experimental results show that MSD methods can effectively avoid the negative effects of SAR image shadows on the corresponding area of the fusion result compared with CS methods, while model-based methods exhibit relatively poor performance. Among all of the fusion methods involved in the comparison, the non-subsampled contourlet transform method (NSCT) presents the best fusion results. In the evaluation using image classification, most experimental results show that the overall classification accuracy after fusion is better than that before fusion. This indicates that optical-SAR fusion can improve land classification, with the gradient transfer fusion method (GTF) yielding the best classification results among all of these fusion methods.

Quantitative Analysis of Meteorite Elements Based on the Multidimensional Scaling–Back Propagation Neural Network Algorithm Combined with Raman Mapping-Assisted Micro-Laser Induced Breakdown Spectroscopy

Meteorites are an essential reference for human exploration of the universe and its cosmic evolution and an essential research object for searching for extraterrestrial life. Ways to quickly identify and screen suspected meteorite samples have become the foundation and prerequisite for research on high-value meteorite samples. Therefore, this paper proposes a Raman mapping-assisted micro-laser induced breakdown spectroscopy (micro-LIBS) technology for field detection of suspected meteorite material composition without sample pre-processing, with a high detection speed and cost-effectiveness, to realize the detection of element composition and molecular structure. Raman mapping carries out multispectral imaging with high spectral resolution of the region of interest. The fusion of Raman mapping and optical microscopy images can provide mineral categories and spatial distribution characteristics in regions of interest. A quantitative analysis model for Fe, Mg, and Na elements was constructed based on the multidimensional scaling–back propagation neural network (MDS-BPNN) algorithm. The determination coefficient of the model test set was better than 0.997, and the root mean square error was better than 0.65. The content of Fe, Mg, and Na elements in the meteorite was preliminarily evaluated, providing a reference for further analysis of element information in spectral image fusion data. The Raman–LIBS combined technology has significant application potential in rapidly evaluating suspected meteorite samples. Without high-end precision instruments or field research, this technology can provide scientists with significant reference value atomic and molecular spectral information. At the same time, this technology can be extended to other petrology research. We offer a fast, efficient, cost-effective, and reliable analysis scheme for reference.

Research on Building AI Learning Dataset for Synthetic Aperture Radar Waterbody Detection through Optical Satellite Image Fusion

For the spatiotemporal analysis of water resources and disasters, water body detection using satellite imagery is crucial. Recently, AI-based methods have been widely employed in water body detection using satellite imagery. To use these AI techniques, a substantial amount of training data is required. When creating training data for water body detection, optical imagery and synthetic aperture radar (SAR) imagery have their respective strengths and weaknesses. To use the advantages of both, this study proposes a water body detection method through the fusion of optical and SAR imagery. The results of the proposed model show an Intersection over Union of 0.612 and an F1 score of 0.759, which is better compared to using either optical or SAR imagery alone. This research presents a method that can easily generate a large amount of water body data, making it promising for use as AI training data for water body detection.

CNN framework for optical image super-resolution and fusion

CNN framework for optical image super-resolution and fusion

Improving the urban features classification accuracy by fusion of optical and radar high spatial resolution images

Improving the urban features classification accuracy by fusion of optical and radar high spatial resolution images

Fault diagnosis of radio frequency circuit using heterogeneous image fusion

Radio frequency (RF) technology covers all fields of life, but with the increase in the size and complexity for circuit devices, traditional contact detection methods have difficulty solving the fault diagnosis problem of electronic circuits. The infrared thermal image fault diagnosis technology can realize perfectly real-time monitoring of the circuit due to it being non-contact and non-destructive and having fast recognition. At present, infrared thermal diagnosis technology can locate the fault but not know what causes the fault. Therefore, we propose a heterogeneous fusion image fault diagnosis method for RF circuits based on image perspective transformation and convolutional neural network (CNN). First, a variety of faults are set on the circuit board, and the faults are numbered; then the optical image, infrared thermal image, and oscilloscope image of circuit board under each fault are obtained by the corresponding instrument and fused heterogeneously. The fused image set is divided into the training set and validation set after enhancement and finally is fed into the CNN for training and classification. The accuracy of the final validation set reached 97.3%, which was 3.4% higher than that of only using infrared thermal image and optical image fusion, 2.7% higher than that of the LeNet5 model, and 1.5% higher than that of Vgg16 model, and the time was shortened by 15 min.

A novel SAR and optical image fusion algorithm based on an improved SPCNN and phase congruency information

ABSTRACT The noise and significant spectral differences lead to severe spectral and spatial information distortions in the fusion result of SAR and optical images. We propose a fusion method based on phase congruency information and an improved, simplified pulse-coupled neural network (PC-SPCNN). The PC-SPCNN method builds the basic fusion framework based on the generalized intensity-hue-saturation transform (GIHS) and nonsubsampled contourlet transform (NSCT). When fusing low-frequency coefficients, a fusion method that couples phase congruency and gain injection is adopted to reduce the spectral distortion caused by nonlinear radiometric differences between images. Meanwhile, an improved, simplified pulse-coupled neural network model is used to fuse the high-frequency coefficients of SAR and optical images. Three groups of multi-source, multi-scale, and multi-scene remote sensing images are used to verify the feasibility of PC-SPCNN and compared with existing fusion algorithms. The results indicate that the PC-SPCNN is superior to existing algorithms in both visual effect and objective evaluation and has better fusion performance.

MCT-Net: Multi-hierarchical cross transformer for hyperspectral and multispectral image fusion

Taking into account the limitations of optical imaging, image acquisition equipment is usually designed to make a trade-off between spatial information and spectral information. Hyperspectral image(HSI) can finely identify and classify imaging objects owing to its rich spectral information, while multispectral image(MSI) can provide fine geometric features because of its sufficient spatial information. Hence, fusing HSI and MSI to achieve information complementarity has become a prevalent manner, which increases the reliability and accuracy of the information obtained. However, unlike traditional optical multi-focus image fusion and pan-sharpening of MSI, existing HSI and MSI fusion methods still face with problems in achieving cross-modality information interaction and lack effective utilization of spatial location information. To solve the above problems and to achieve more effective information integration between HSI and MSI, this paper proposes a novel multi-hierarchical cross transformer for hyperspectral and multispectral image fusion (MCT-Net). The proposed MCT-Net consists of two components: (1) a multi-hierarchical cross-modality interacting module (MCIM), which first extracts the deep multi-scale features of HSI and MSI, and then performs cross-modality information interaction between them at identical scales by applying a multi-hierarchical cross transformer (MCT), to reconstruct the spectral information lacking in MSI and the spatial information lacking in HSI; (2) a feature aggregation reconstruction module (FARM) which combines features from MCIM, uses strip convolution to further restore edge features, and reconstructs the fusion results through cascaded upsampling. We conduct comparative experiments on five mainstream HSI datasets to prove the effectiveness and superiority of the proposed method, including the Pavia Center, Pavia University, Urban, Botswana, and Washington DC Mall. For instance, on the Washington DC Mall dataset, compared with the state-of-the-art(SOTA) method in the comparison algorithms, our method improves PSNR by 18.52% and reduces RMSE, ERGAS and SAM by 56.63%, 56.90% and 58.58%, respectively. The source code for MCT-Net can be downloaded from https://github.com/wxy11-27/MCT-Net.

MDAS: a new multimodal benchmark dataset for remote sensing

Abstract. In Earth observation, multimodal data fusion is an intuitive strategy to break the limitation of individual data. Complementary physical contents of data sources allow comprehensive and precise information retrieval. With current satellite missions, such as ESA Copernicus programme, various data will be accessible at an affordable cost. Future applications will have many options for data sources. Such a privilege can be beneficial only if algorithms are ready to work with various data sources. However, current data fusion studies mostly focus on the fusion of two data sources. There are two reasons; first, different combinations of data sources face different scientific challenges. For example, the fusion of synthetic aperture radar (SAR) data and optical images needs to handle the geometric difference, while the fusion of hyperspectral and multispectral images deals with different resolutions on spatial and spectral domains. Second, nowadays, it is still both financially and labour expensive to acquire multiple data sources for the same region at the same time. In this paper, we provide the community with a benchmark multimodal data set, MDAS, for the city of Augsburg, Germany. MDAS includes synthetic aperture radar data, multispectral image, hyperspectral image, digital surface model (DSM), and geographic information system (GIS) data. All these data are collected on the same date, 7 May 2018. MDAS is a new benchmark data set that provides researchers rich options on data selections. In this paper, we run experiments for three typical remote sensing applications, namely, resolution enhancement, spectral unmixing, and land cover classification, on MDAS data set. Our experiments demonstrate the performance of representative state-of-the-art algorithms whose outcomes can serve as baselines for further studies. The dataset is publicly available at https://doi.org/10.14459/2022mp1657312 (Hu et al., 2022a) and the code (including the pre-trained models) at https://doi.org/10.5281/zenodo.7428215 (Hu et al., 2022b).

Fusion and Classification of SAR and Optical Data Using Multi-Image Color Components with Differential Gradients

This paper proposes a gradient-based data fusion and classification approach for Synthetic Aperture Radar (SAR) and optical image. This method is used to intuitively reflect the boundaries and edges of land cover classes present in the dataset. For the fusion of SAR and optical images, Sentinel 1A and Sentinel 2B data covering Central State Farm in Hissar (India) was used. The major agricultural crops grown in this area include paddy, maize, cotton, and pulses during kharif (summer) and wheat, sugarcane, mustard, gram, and peas during rabi (winter) seasons. The gradient method using a Sobel operator and color components for three directions (i.e., x, y, and z) are used for image fusion. To judge the quality of fused image, several fusion metrics are calculated. After obtaining the resultant fused image, gradient based classification methods, including Stochastic Gradient Descent Classifier, Stochastic Gradient Boosting Classifier, and Extreme Gradient Boosting Classifier, are used for the final classification. The classification accuracy is represented using overall classification accuracy and kappa value. A comparison of classification results indicates a better performance by the Extreme Gradient Boosting Classifier.