Image Fusion Research Articles

Recognition of small water bodies under complex terrain based on SAR and optical image fusion algorithm

Understanding the spatial and temporal distribution of small water bodies is essential for managing water resources, crafting conservation policies, and preserving watershed ecosystems and biodiversity. However, existing studies often rely on a single remote sensing data source (optical or microwave), focusing on large-scale, flat areas and lacking comprehensive monitoring of small water bodies in complex terrain. Therefore, considering the complementary advantages of multisource remote sensing (multispectral and SAR), this paper proposes a multispectral and SAR fusion algorithm, named Multispectral and SAR Fusion algorithm (MASF), to better capture the remote sensing characteristics of small water bodies in complex areas. Based on this, a dataset containing spectral, texture, and geometric features is constructed, and multi-scale segmentation and random forest algorithms are applied for identification of small water bodies in complex terrain. The results demonstrate that the proposed fusion algorithm MASF exhibits minimal spectral distortion (SAM < 3.5, ERGAS <21, RMSE <0.01) and robust spatial feature enhancement (PSNR >40, SSIM >0.999, CC > 0.99). The Overall Accuracy (OA) and Kappa coefficients for both experimental areas surpassed 0.9. For rivers and reservoirs, both Producer's Accuracy (PA) and User's Accuracy (UA) exceeded 0.9. The UA for agricultural ponds exceeded 0.8. Comparative analysis with three other types of water-related data products shows that the freshwater identification results in this study have certain advantages in local small water bodies. Our research holds significant implications for the utilization of water resources in mountainous areas, prevention and control of floods and floods, as well as the development of aquaculture industry.

Semantic attention-based heterogeneous feature aggregation network for image fusion

Infrared and visible image fusion aims to generate a comprehensive image that retains both salient targets of the infrared image and texture details of the visible image. However, existing methods overlook the differences in attention to semantic information among different fused images. To address this issue, we propose a semantic attention-based heterogeneous feature aggregation network for image fusion. The key component of our network is the semantic attention-based fusion module, which leverages the weights derived from semantic feature maps to dynamically adjust the significance of various semantic objects within the fusion feature maps. By using semantic weights as guidance, our fusion process concentrates on regions with crucial semantics, resulting in a more focused fusion that preserves rich semantic information. Moreover, we propose an innovative component called the attentive dense block. This block effectively filters out irrelevant features during extraction, accentuates essential features to their maximum potential, and enhances the visual quality of the fused images. Importantly, our network demonstrates strong generalization capabilities. Extensive experiments validate the superiority of our proposed network over current state-of-the-art techniques in terms of both visual appeal and semantics-driven evaluation.

Infrared and visible image fusion and detection based on interactive training strategy and feature filter extraction module

The infrared and visible image fusion-detection technique aims to expand the observation wavelength range to overcome harsh environmental conditions such as low light, smoke, heavy fog, etc., and improve the reliability and robustness of target detection. However, the current fusion-detection method suffers from problems of the joint optimization of the fusion-detection model and irrelevant feature interference. To address these problems, we propose an infrared and visible image fusion-detection network based on an interactive training strategy (ITS) and a feature filter extraction module (FFEM). The ITS constructs the closed-loop optimization for the fusion-detection method, which can mutually enhance the performance of the fusion and detection model through mutual supervision. The FFEM is designed to acquire detection-conducive features from visible and infrared images. Specifically, we define filtering weights based on the covariance values of features and use these weights to enhance relevant features and soft redundant and irrelevant features (e.g., noise). Quantitative and qualitative experimental results demonstrate that the proposed method achieves competitive performance in target detection accuracy compared with other state-of-the-art fusion-detection methods.

A self-supervised fusion for infrared and visible images via multi-level contrastive auto-encoding

The objective of infrared and visible image fusion is to integrate the features of source images and acquire a fused image that highlights foreground objects and preserves background details. Nonetheless, due to ignoring the mutual information interaction between source images, prevailing image fusion methods always encounter challenges related to incomplete or redundant feature extraction. For this purpose, this study presents a self-supervised fusion framework,employing multi-level contrastive auto-encoding to realize the infrared and visible image fusion (IVIF) task, referred to as SS-MCAE. The SS-MCAE consists of a coarse-to-fine feature extraction network (CFEN) and a fusion network (FN), where CFEN contains a content-aware enhancement module (CEM) and a multi-level contrastive learning autoencoder (MCLA). Specifically, CFEN employs CEM to adaptively enhance brightness information and texture details from source images and the MCLA is introduced to refine and extract these features hierarchically. Within contrastive learning, a neoteric sample generator (SG) is introduced for constructing positive and negative samples, aiming to maximize feature extraction while diminishing redundant information. Furthermore, an innovative perceptual loss is developed to retain original features and guide the image reconstruction, establishing a more reliable relationship between extracted features and source images. Comprehensive experiments reveal that our proposed SS-MCAE is superior to current approaches in both visual effect and quantitative analysis.

UPGAN: An Unsupervised Generative Adversarial Network Based on U-Shaped Structure for Pansharpening

Pansharpening is the fusion of panchromatic images and multispectral images to obtain images with high spatial resolution and high spectral resolution, which have a wide range of applications. At present, methods based on deep learning can fit the nonlinear features of images and achieve excellent image quality; however, the images generated with supervised learning approaches lack real-world applicability. Therefore, in this study, we propose an unsupervised pansharpening method based on a generative adversarial network. Considering the fine tubular structures in remote sensing images, a dense connection attention module is designed based on dynamic snake convolution to recover the details of spatial information. In the stage of image fusion, the fusion of features in groups is applied through the cross-scale attention fusion module. Moreover, skip layers are implemented at different scales to integrate significant information, thus improving the objective index values and visual appearance. The loss function contains four constraints, allowing the model to be effectively trained without reference images. The experimental results demonstrate that the proposed method outperforms other widely accepted state-of-the-art methods on the QuickBird and WorldView2 data sets.

DSR-Diff: Depth map super-resolution with diffusion model

Color-guided depth map super-resolution (CDSR) improves the spatial resolution of a low-quality depth map with the corresponding high-quality color map, benefiting various applications such as 3D reconstruction, virtual reality, and augmented reality. While conventional CDSR methods typically rely on convolutional neural networks or transformers, diffusion models (DMs) have demonstrated notable effectiveness in high-level vision tasks. In this work, we present a novel CDSR paradigm that utilizes a diffusion model within the latent space to generate guidance for depth map super-resolution. The proposed method comprises a guidance generation network (GGN), a depth map super-resolution network (DSRN), and a guidance recovery network (GRN). The GGN is specifically designed to generate the guidance while managing its compactness. Additionally, we integrate a simple but effective feature fusion module and a transformer-style feature extraction module into the DSRN, enabling it to leverage the guiding prior in the extraction, fusion, and reconstruction of multi-model images. Taking into account both accuracy and efficiency, our proposed method has shown superior performance in extensive experiments compared to state-of-the-art methods. Specifically, our proposed DSR-Diff demonstrates remarkable improvements of 18.7% and 15.9% in terms of RMSE at scales 8 and 16 on the NYU v2 dataset while saving nearly 30% of the runtime, compared to existing SOTA method. Our codes will be made available at https://github.com/shiyuan7/DSR-Diff.

Infrared and visible image fusion based on FUDPCNN and gravitational force operator

Infrared and visible image fusion based on FUDPCNN and gravitational force operator

A multi‐focus image fusion network deployed in smart city target detection

AbstractIn the global monitoring of smart cities, the demands of global object detection systems based on cloud and fog computing in intelligent systems can be satisfied by photographs with globally recognized properties. Nevertheless, conventional techniques are constrained by the imaging depth of field and can produce artefacts or indistinct borders, which can be disastrous for accurately detecting the object. In light of this, this paper proposes an artificial intelligence‐based gradient learning network that gathers and enhances domain information at different sizes in order to produce globally focused fusion results. Gradient features, which provide a lot of boundary information, can eliminate the problem of border artefacts and blur in multi‐focus fusion. The multiple‐receptive module (MRM) facilitates effective information sharing and enables the capture of object properties at different scales. In addition, with the assistance of the global enhancement module (GEM), the network can effectively combine the scale features and gradient data from various receptive fields and reinforce the features to provide precise decision maps. Numerous experiments have demonstrated that our approach outperforms the seven most sophisticated algorithms currently in use.

UCP2-ACGAN: An adaptive condition GAN guided by U-shaped context perceptual processor for PET/CT images fusion

In order to better highlight the lesion area information and edge detailed information of the fused images, an adaptive condition Generative Adversarial Network (GAN) guided by U-shaped context perceptual processor (UCP2-ACGAN) for Positron Emission Tomography (PET) and Computed Tomography (CT) images fusion is proposed in this paper. Firstly, the network architecture of single generator and dual relative discriminators with weights sharing is used in UCP2-ACGAN. Secondly, a multi-granularity adaptive feature extraction module from coarse to fine (MGAFE) is proposed in generator, it is used to extract the valid information contained in the source images. Thirdly, a dual branch feature interactive fusion module guided by U-shaped context perceptual processor (UCP2-DBFIF) is proposed in generator. The context perceptual feature maps are obtained by U-shaped context perceptual processor (UCP2) and multiply them with the feature extraction results as the guiding conditions in the fusion process. In addition, in order to supplement the effective information lost during the feature fusion process, this paper converges the feature information of different granularity during the feature extraction stage. Finally, compared to the best of the 6 comparison methods, the results of comparison experiments show that UCP2-ACGAN is effective on several evaluation metrics values. For example, on the evaluation metrics of Average Gradient (AG), Edge Intensity (EI), Spatial Frequency (SF), Visual Information Fidelity (VIF) and Mutual Information (MI), the metrics values of CT mediastinal window images and PET fused images average increased by 30.74%, 29.35%, 40.24%, 35.47% and 26.18%.

Mitigating human fall injuries: A novel system utilizing 3D 4-stream convolutional neural networks and image fusion

Unintentional human falls, especially in seniors, lead to serious injuries, fatalities, and reduced standard of life. Vision-based fall detection methods have demonstrated their usefulness in timely fall response, helping to lessen such injuries. This paper presents an automated vision-based fall detection system that triggers immediate fall reporting. By incorporating human segmentation and image fusion in the pre-processing stage, the system enhances the accuracy of human action classification, thereby ensuring precise fall alerts. It further employs the innovative 4-stream 3D convolutional neural network (4S-3DCNN) model to learn different but consecutive spatial and temporal features. The system processes video input or live surveillance, segmenting human presence every 32 frames using a fine-tuned deep-learning model and applying a three-level image fusion to accentuate movement differences. This technique produces four pre-processed images, input to the 4S-3DCNN model for classification. Consecutive detection of “Falling” and “Fallen” actions triggers an alert for immediate intervention. The original 4S-3DCNN model is an end-to-end trained deep learning model with a fully connected layer serving as a classifier. The research also evaluates the performance of combining the 4S-3DCNN model with Autoencoders and Support Vector Machines (SVM) networks as classifiers. The SVM classifier demonstrated ideal fall detection performance with 100% accuracy using the MCFD, URFD, and Le2i FDD datasets. The proposed system is vital for detecting and preventing falls and reducing healthcare expenses and productivity losses.

Image Fusion of Multi-Temporal Images of Eyes Feature using Hybrid Approach of PCA and SWT

A wide variety of information is represented by an image feature, which may be applied in various applications such as mage fusion, video processing, medical diagnosis, traffic safety monitoring, visual surveillance, feature matching, image segmentation, pattern matching, person identification, sentiment analysis, human computer interaction and many more fields. This paper focused on multi-temporal feature extraction of eyes feature from human face images captured in two different time slots to reduce the semantic gap from images and to improve the image quality by image fusion using PCA, SWT and hybrid approach of PCA and SWT algorithm. In this paper, comparative study of multi-temporal image fusion using Principle Component Analysis (PCA), Stationary Wavelet Transform (SWT) algorithms and hybrid approach of PCA and SWT are employed and its experimental results are evaluated with its performance analysis. Image fusion performance is compared based on eight quantitative quality measures as SSIM, MSE, NAE, CC , SC , AD, SD and MI.The outcomes of comparison show that employing the hybrid approach of PCA+SWT transform can improve image fusion performance. The applicability of this work approach may have several uses when the utilization of human facial features is feasible.

Tongue image fusion and analysis of thermal and visible images in diabetes mellitus using machine learning techniques

The study aimed to achieve the following objectives: (1) to perform the fusion of thermal and visible tongue images with various fusion rules of discrete wavelet transform (DWT) to classify diabetes and normal subjects; (2) to obtain the statistical features in the required region of interest from the tongue image before and after fusion; (3) to distinguish the healthy and diabetes using fused tongue images based on deep and machine learning algorithms. The study participants comprised of 80 normal subjects and age- and sex-matched 80 diabetes patients. The biochemical tests such as fasting glucose, postprandial, Hba1c are taken for all the participants. The visible and thermal tongue images are acquired using digital single lens reference camera and thermal infrared cameras, respectively. The digital and thermal tongue images are fused based on the wavelet transform method. Then Gray level co-occurrence matrix features are extracted individually from the visible, thermal, and fused tongue images. The machine learning classifiers and deep learning networks such as VGG16 and ResNet50 was used to classify the normal and diabetes mellitus. Image quality metrics are implemented to compare the classifiers’ performance before and after fusion. Support vector machine outperformed the machine learning classifiers, well after fusion with an accuracy of 88.12% compared to before the fusion process (Thermal-84.37%; Visible-63.1%). VGG16 produced the classification accuracy of 94.37% after fusion and attained 90.62% and 85% before fusion of individual thermal and visible tongue images, respectively. Therefore, this study results indicates that fused tongue images might be used as a non-contact elemental tool for pre-screening type II diabetes mellitus.

Balanced spatio-spectral feature extraction for hyperspectral and multispectral image fusion

Although remote sensing sensors acquire hyperspectral images (HSI) with outstanding spectral resolution, hardware limitations prevent them from obtaining HSI with superior spatial resolution. Having high-resolution multispectral images (MSI) from the same scene has sparked the concept of HSI-MSI fusion to achieve high-resolution HSI. Recently, deep learning algorithms have shown excellent performance in HSI-MSI fusion. Deep learning algorithms can automatically extract the priors latent in the input data and reconstruct the output image. However, their performance depends on the training data’s inherent characteristics. HSI volume is generally significantly larger than MSI volume. While the high-resolution MSI provides richer spatial information, its contribution to fusion is lower due to its smaller volume. To address this volume discrepancy, a deep convolutional neural network named BASFE is proposed. This network employs a two-branch structure that balances the extracted spatio-spectral features from both HSI and MSI input images, enabling effective feature fusion. Therefore, better efficiency can be achieved by using simpler structures. Experiments showed that balancing the data improves the model’s performance significantly. Furthermore, to improve efficiency and reduce complexity and computational burden, the spatio-spectral feature extraction residual modules are embedded in a dense architecture to jointly extract these features. Comprehensive experiments on five challenging datasets show that BASFE surpasses state-of-the-art algorithms. Compared to the first- and second-best competing methods, the proposed method achieves an average improvement of 1.58 dB and 3.68 dB in peak signal-to-noise ratio (PSNR) across the five studied datasets. The source code of BASFE is available in https://github.com/rajaei-arash/BASFE_hyperspectral_fusion.git.

Dual-channel aperture-splitting snapshot spectral imaging detection system

The contemporary spectral imaging detection systems commonly employed, such as pushbroom and stare systems, often necessitate motion-based imaging mechanisms such as scanning motors. This reliance on motion renders the imaging process vulnerable to platform vibrations, resulting in intricate post-image correction procedures and precluding dynamic target detection. Consequently, the advent of snapshot spectral imaging detection systems has ensued. Currently, there are significant challenges in the miniaturization design and rapid data acquisition aspects of snapshot spectral imaging systems. In this study, linear variable filters were employed as spectral components, and, through optical system simulation and design, the design of the nonspherical monolithic lens system and telescope system in the dual-channel aperture-splitting snapshot spectral imaging detection system were separately completed. The spectral range covered 400–1000 nm, with a spectral resolution of 27.3 nm, and each channel had a spatial sampling of 409×409 pixels. Additionally, based on the optical system design results, the system structure design and assembly were completed. Performance testing and preliminary spectral image fusion research were conducted on the assembled prototype. The instrument demonstrated excellent spectral imaging performance, thereby enhancing the efficiency of spectral imaging detection in snapshot spectral imaging.

DCFNet: Infrared and Visible Image Fusion Network Based on Discrete Wavelet Transform and Convolutional Neural Network.

Aiming to address the issues of missing detailed information, the blurring of significant target information, and poor visual effects in current image fusion algorithms, this paper proposes an infrared and visible-light image fusion algorithm based on discrete wavelet transform and convolutional neural networks. Our backbone network is an autoencoder. A DWT layer is embedded in the encoder to optimize frequency-domain feature extraction and prevent information loss, and a bottleneck residual block and a coordinate attention mechanism are introduced to enhance the ability to capture and characterize the low- and high-frequency feature information; an IDWT layer is embedded in the decoder to achieve the feature reconstruction of the fused frequencies; the fusion strategy adopts the l1-norm fusion strategy to integrate the encoder's output frequency mapping features; a weighted loss containing pixel loss, gradient loss, and structural loss is constructed for optimizing network training. DWT decomposes the image into sub-bands at different scales, including low-frequency sub-bands and high-frequency sub-bands. The low-frequency sub-bands contain the structural information of the image, which corresponds to the important target information, while the high-frequency sub-bands contain the detail information, such as edge and texture information. Through IDWT, the low-frequency sub-bands that contain important target information are synthesized with the high-frequency sub-bands that enhance the details, ensuring that the important target information and texture details are clearly visible in the reconstructed image. The whole process is able to reconstruct the information of different frequency sub-bands back into the image non-destructively, so that the fused image appears natural and harmonious visually. Experimental results on public datasets show that the fusion algorithm performs well according to both subjective and objective evaluation criteria and that the fused image is clearer and contains more scene information, which verifies the effectiveness of the algorithm, and the results of the generalization experiments also show that our network has good generalization ability.

Bidirectional Feature Fusion and Enhanced Alignment based Multimodal Semantic Segmentation for Remote Sensing Images

Image–text multimodal deep semantic segmentation leverages the fusion and alignment of image and text information and provides more prior knowledge for segmentation tasks. It is worth exploring image–text multimodal semantic segmentation for remote sensing images. In this paper, we propose a bidirectional feature fusion and enhanced alignment-based multimodal semantic segmentation model (BEMSeg) for remote sensing images. Specifically, BEMSeg first extracts image and text features by image and text encoders, respectively, and then the features are provided for fusion and alignment to obtain complementary multimodal feature representation. Secondly, a bidirectional feature fusion module is proposed, which employs self-attention and cross-attention to adaptively fuse image and text features of different modalities, thus reducing the differences between multimodal features. For multimodal feature alignment, the similarity between the image pixel features and text features is computed to obtain a pixel–text score map. Thirdly, we propose a category-based pixel-level contrastive learning on the score map to reduce the differences among the same category’s pixels and increase the differences among the different categories’ pixels, thereby enhancing the alignment effect. Additionally, a positive and negative sample selection strategy based on different images is explored during contrastive learning. Averaging pixel values across different training images for each category to set positive and negative samples compares global pixel information while also limiting sample quantity and reducing computational costs. Finally, the fused image features and aligned pixel–text score map are concatenated and fed into the decoder to predict the segmentation results. Experimental results on the ISPRS Potsdam, Vaihingen, and LoveDA datasets demonstrate that BEMSeg is superior to comparison methods on the Potsdam and Vaihingen datasets, with improvements in mIoU ranging from 0.57% to 5.59% and 0.48% to 6.15%, and compared with Transformer-based methods, BEMSeg also performs competitively on LoveDA dataset with improvements in mIoU ranging from 0.37% to 7.14%.

PGGNet: Pyramid gradual-guidance network for RGB-D indoor scene semantic segmentation

In RGB-D (red–green–blue and depth) scene semantic segmentation, depth maps provide rich spatial information to RGB images to achieve high performance. However, properly aggregating depth information and reducing noise and information loss during feature encoding after fusion are challenging aspects in scene semantic segmentation. To overcome these problems, we propose a pyramid gradual-guidance network for RGB-D indoor scene semantic segmentation. First, the quality of depth information is improved by a modality-enhancement fusion module and RGB image fusion. Then, the representation of semantic information is improved by multiscale operations. The two resulting adjacent features are used in a feature refinement module with an attention mechanism to extract semantic information. The features from adjacent modules are successively used to form an encoding pyramid, which can substantially reduce information loss and thereby ensure information integrity. Finally, we gradually integrate features at the same scale obtained from the encoding pyramid during decoding to obtain high-quality semantic segmentation. Experimental results obtained from two commonly used indoor scene datasets demonstrate that the proposed pyramid gradual-guidance network attains the highest level of performance in semantic segmentation, as compared to other existing methods.

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer.

The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.

An improved hybrid fusion of noisy medical images using differential evolution-based artificial rabbits optimization algorithm

An improved hybrid fusion of noisy medical images using differential evolution-based artificial rabbits optimization algorithm

Underwater image enhancement via color correction and multi-feature image fusion

Abstract The light attenuation underwater causes the actual underwater images to suffer from color cast, low contrast, and weak illumination. To address these issues, an effective fusion-based method is proposed, which realizes color correction, brightness adjustment, contrast, and detail enhancement of underwater images. Concretely, we first design an adaptive color correction method via dominant color channel judgment and lower color channel compensation. Then, we detect the brightness of each input image and propose a gamma correction function based on the gradient of the cumulative histogram to adjust the brightness of the low-light images. Subsequently, global histogram stretching and adaptive fractional differentiation techniques are employed to process the brightness-adjusted image, and then the global contrast-enhanced version and detail-enhanced version are generated respectively. To integrate the advantages of both versions, a channel fusion method based on the Lab color space is used to fuse the luminance and color of the two versions separately. The experimental results demonstrate the effectiveness of the proposed method in improving the color and illumination of underwater images, as well as enhancing the clarity of images. Moreover, the testing results on multiple datasets validate the excellent stability of this method.&amp;#xD;