Image Fusion Model Research Articles

High spatial–temporal image fusion model for retrieving aerosol optical depth based on top-of-atmosphere reflectance

High spatial–temporal image fusion model for retrieving aerosol optical depth based on top-of-atmosphere reflectance

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

A Task-Guided, Implicitly-Searched and Meta-Initialized Deep Model for Image Fusion.

Image fusion plays a key role in a variety of multi-sensor-based vision systems, especially for enhancing visual quality and/or extracting aggregated features for perception. However, most existing methods just consider image fusion as an individual task, thus ignoring its underlying relationship with these downstream vision problems. Furthermore, designing proper fusion architectures often requires huge engineering labor. It also lacks mechanisms to improve the flexibility and generalization ability of current fusion approaches. To mitigate these issues, we establish a Task-guided, Implicit-searched and Meta-initialized (TIM) deep model to address the image fusion problem in a challenging real-world scenario. Specifically, we first propose a constrained strategy to incorporate information from downstream tasks to guide the unsupervised learning process of image fusion. Within this framework, we then design an implicit search scheme to automatically discover compact architectures for our fusion model with high efficiency. In addition, a pretext meta initialization technique is introduced to leverage divergence fusion data to support fast adaptation for different kinds of image fusion tasks. Qualitative and quantitative experimental results on different categories of image fusion problems and related downstream tasks (e.g., visual enhancement and semantic understanding) substantiate the flexibility and effectiveness of our TIM.

FusionOC: Research on optimal control method for infrared and visible light image fusion

Infrared and visible light image fusion can solve the limitations of single-type visual sensors and can boost the target detection performance. However, since the traditional fusion strategy lacks the controllability and feedback mechanism, the fusion model cannot precisely perceive the relationship between the requirements of the fusion task, the fused image quality, and the source image features. To this end, this paper establishes a fusion model based on the optimal controlled object and control mode called FusionOC. This method establishes two types of mathematical models of the controlled objects by verifying the factors and conflicts affecting the quality of the fused image. It combines the image fusion model with the quality evaluation function to determine the two control factors separately. At the same time, two proportional-integral-derivative (PID) control and regulation modes based on the backpropagation (BP) neural network are designed according to the control factor characteristics. The fusion system can adaptively select the regulation mode to regulate the control factor according to the user requirements or the task to make the fusion system perceive the connection between the fusion task and the result. Besides, the fusion model employs the feedback mechanism of the control system to perceive the feature difference between the fusion result and the source image, realize the guidance of the source image feature to the entire fusion process, and improve the fusion algorithm's generalization ability and intelligence level when handling different fusion tasks. Experimental results on multiple public datasets demonstrate the advantages of FusionOC over advanced methods. Meanwhile, the benefits of our fusion results in object detection tasks have been demonstrated.

Night Lighting Fault Detection Based on Improved YOLOv5

Night lighting is essential for urban life, and the occurrence of faults can significantly affect the presentation of lighting effects. Many reasons can cause lighting faults, including the damage of lamps and circuits, and the typical manifestation of the faults is that the lights do not light up. The current troubleshooting mainly relies on artificial visual inspection, making detecting faults difficult and time-consuming. Therefore, it is necessary to introduce technical means to detect lighting faults. However, current research on lighting fault detection mainly focuses on using non-visual methods such as sensor data analysis, which has the disadvantages of having a high cost and difficulty adapting to large-scale fault detection. Therefore, this study mainly focuses on solving the problem of the automatic detection of night lighting faults using machine vision methods, especially object detection methods. Based on the YOLOv5 model, two data fusion models have been developed based on the characteristics of lighting fault detection inverse problems: YOLOv5 Channel Concatenation and YOLOv5 Image Fusion. Based on the dataset obtained from the developed automatic image collection and annotation system, the training and evaluation of these three models, including the original YOLOv5, YOLOv5 Channel Concatenation, and YOLOv5 Image Fusion, have been completed. Research has found that applying complete lighting images is essential for the problem of lighting fault detection. The developed Image Fusion model can effectively fuse information and accurately detect the occurrence and area of faults, with a mAP value of 0.984. This study is expected to play an essential role in the intelligent development of urban night lighting.

DCFusion: Difference correlation-driven fusion mechanism of infrared and visible images

In end-to-end image fusion models, the loss function significantly impacts performance. However, most loss functions treat salient and background regions in source images equally, failing to distinguish complementary areas in multimodal images. This limits the model’s ability to effectively integrate information from these regions. Therefore, we propose difference correlation-driven fusion mechanism of infrared and visible images, which called DCFusion. Specifically, the model utilizes a dual-branch interactive network that dynamically fuses cross-modal multi-scale complementary information through element-wise multiplication, effectively integrating region-specific information. We introduce a two-stage method for generating salient target masks that adaptively focus on high-contrast regions in infrared images by analyzing pixel contrasts in local areas. Furthermore, we utilize the salient target masks to create heterogeneous images and design the LSCD loss function to minimize the information gap between the heterogeneous images and the fused image, thereby enhancing the model’s interpretability. Experiments on the RoadScene and TNO datasets show that DCFusion surpasses with existing representativity fusion approaches, achieving state-of-the-art performance in both subjective visual and objective evaluations. Our code will be publicly available at https://github.com/MinLila/DCFusion.

Self-adaptive 2D[sbnd]3D image fusion for automated pixel-level pavement crack detection

Current 2D and 3D image-based crack detection methods in transportation infrastructure often struggle with noise robustness and feature diversity. To overcome these challenges, the paper use CSF-CrackNet, a self-adaptive 2D3D image fusion model utilizes channel and spatial modules for automated pavement crack segmentation. CSF-CrackNet consists of four parts: feature enhanced and field sensing (FEFS) module, channel module, spatial module, and semantic segmentation module. A multi-feature image dataset was established using a vehicle-mounted 3D imaging system, including color images, depth images, and color-depth overlapped images. Results show that the mean intersection over union (mIOU) of most models under the CSF-CrackNet framework can be increased to above 80 %. Compared with original RGB and depth images, the average mIOU increases with image fusion by 10 % and 5 %, respectively. The ablation experiment and weight significance analysis further demonstrate that CSF-CrackNet can significantly improve semantic segmentation performance by balancing information between 2D and 3D images.

EDOM-MFIF: an end-to-end decision optimization model for multi-focus image fusion

EDOM-MFIF: an end-to-end decision optimization model for multi-focus image fusion

Enhanced multimodal medical image fusion via modified DWT with arithmetic optimization algorithm

Medical image fusion (MIF) techniques are proficient in combining medical images in distinct morphologies to obtain a reliable medical analysis. A single modality image could not offer adequate data for an accurate analysis. Therefore, a novel multimodal MIF-based artificial intelligence (AI) method has been presented. MIF approaches fuse multimodal medical images for exact and reliable medical recognition. Multimodal MIF improves diagnostic accuracy and clinical decision-making by combining complementary data in different imaging modalities. This article presents a new multimodal medical image fusion model utilizing Modified DWT with an Arithmetic Optimization Algorithm (MMIF-MDWTAOA) approach. The MMIF-MDWTAOA approach aims to generate a fused image with the significant details and features from each modality, leading to an elaborated depiction for precise interpretation by medical experts. The bilateral filtering (BF) approach is primarily employed for noise elimination. Next, the image decomposition process uses a modified discrete wavelet transform (MDWT) approach. However, the approximation coefficient of modality_1 and the detailed coefficient of modality_2 can be fused interchangeably. Furthermore, a fusion rule is derived from combining the multimodality data, and the AOA model is enforced to ensure the optimum selection of the fusion rule parameters. A sequence of simulations is accomplished to validate the enhanced output of the MMIF-MDWTAOA technique. The investigational validation of the MMIF-MDWTAOA technique showed the highest entropy values of 7.568 and 7.741 bits/pixel over other approaches.

LFDT-Fusion: A latent feature-guided diffusion Transformer model for general image fusion

For image fusion tasks, it is inefficient for the diffusion model to iterate multiple times on the original resolution image for feature mapping. To address this issue, this paper proposes an efficient latent feature-guided diffusion model for general image fusion. The model consists of a pixel space autoencoder and a compact Transformer-based diffusion network. Specifically, the pixel space autoencoder is a novel UNet-based latent diffusion strategy that compresses inputs into a low-resolution latent space through downsampling. Simultaneously, skip connections transfer multiscale intermediate features from the encoder to the decoder for decoding, preserving the high-resolution information of the original input. Compared to the existing VAE-GAN-based latent diffusion strategy, the proposed UNet-based strategy is significantly more stable and generates highly detailed images without adversarial optimization. The Transformer-based diffusion network consists of a denoising network and a fusion head. The former captures long-range diffusion dependencies and learns hierarchical diffusion representations, while the latter facilitates diffusion feature interactions to comprehend complex cross-domain information. Moreover, improvements to the diffusion model in noise level, denoising steps, and sampler selection have yielded superior fusion performance across six image fusion tasks. The proposed method illustrates qualitative and quantitative advantages, as evidenced by experimental results in both public datasets and industrial environments. The code is available at: https://github.com/BOYang-pro/LFDT-Fusion.

Research on metal surface specular removal algorithm based on unsupervised learning

The highlights on the surface of metal materials can seriously destroy the continuity of the image, produce certain false edges, and cause the texture details in the highlights area to weaken or even disappear, which interferes with the subsequent operations such as surface region segmentation and defect detection. Aiming at the low efficiency, high loss, easy distortion and difficult calibration of metal highlight data, an unsupervised perceptual enhancement network model based on the convolutional neural network (CNN) is proposed. Firstly, the method of generating antagonism is used to generate a large number of metal images with high-light feature information, which is used to increase the number of high-light metal image data sets in the training set. Secondly, a detail enhance model(DEM) and a color enhance model(CEM) are introduced into the context aggregation network to improve the feature detail retention rate in the low resolution weight graphs. Finally, the multi-scale structural similarity function is used to replace the original structural similarity function to solve the insensitive detail when the image size is too large. Experiments show that comparing with other multi-exposure image fusion models, the present model can improve the evaluation index of mutual information and average gradient of fused images by about 10%, and can retain more texture feature information.

Mapping Crop Evapotranspiration by Combining the Unmixing and Weight Image Fusion Methods

The demand for freshwater is increasing with population growth and rapid socio-economic development. It is more and more important for refined irrigation water management to conduct research on crop evapotranspiration (ET) data with a high spatiotemporal resolution in agricultural regions. We propose the unmixing–weight ET image fusion model (UWET), which integrates the advantages of the unmixing method in spatial downscaling and the weight-based method in temporal prediction to produce daily ET maps with a high spatial resolution. The Landsat-ET and MODIS-ET datasets for the UWET fusion data are retrieved from Landsat and MODIS images based on the surface energy balance model. The UWET model considers the effects of crop phenology, precipitation, and land cover in the process of the ET image fusion. The precision evaluation is conducted on the UWET results, and the measured ET values are monitored by eddy covariance at the Luancheng station, with average MAE values of 0.57 mm/day. The image results of UWET show fine spatial details and capture the dynamic ET changes. The seasonal ET values of winter wheat from the ET map mainly range from 350 to 660 mm in 2019–2020 and from 300 to 620 mm in 2020–2021. The average seasonal ET in 2019–2020 is 499.89 mm, and in 2020–2021, it is 459.44 mm. The performance of UWET is compared with two other fusion models: the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and the Spatial and Temporal Reflectance Unmixing Model (STRUM). UWET performs better in the spatial details than the STARFM and is better in the temporal characteristics than the STRUM. The results indicate that UWET is suitable for generating ET products with a high spatial–temporal resolution in agricultural regions.

A Security Study of Multimodel Artificial Intelligence System: Adaptive Retention Attack for Object Detection System with Multifocus Image Fusion Model

Image preprocessing models are usually employed as the preceding operations of high‐level vision tasks to improve the performance. The adversarial attack technology makes both these models face severe challenges. Prior research is focused solely on attacking single object detection models, without considering the impact of the preprocessing models (multifocus image fusion) on adversarial perturbations within the object detection system. Multifocus image fusion models work in conjunction with the object detection models to enhance the quality of the images and improve the capability of object detection system. Herein, the problem of attacking object detection system that utilizes multifocus image fusion as its preprocessing models is addressed. To retain the attack capabilities of adversarial samples against as many perturbations as possible, new attack method called adaptive retention attack (ARA) is proposed. Additionally, adversarial perturbations concentration mechanism and image selection mechanism, which, respectively, enhance the transferability and attack capability of ARA‐generated adversarial samples. Extensive experiments have demonstrated the feasibility of the ARA. The results confirm that the ARA method can successfully bypass multifocus image fusion models to attack the object detection model.

Leveraging a self-adaptive mean teacher model for semi-supervised multi-exposure image fusion

Deep learning-based methods have recently shown remarkable advancements in multi-exposure image fusion (MEF), demonstrating significant achievements in improving the fusion quality. Despite their success, the majority of reference images in MEF are artificially generated, inevitably introducing a portion of low-quality ones. Existing methods either utilize these mixed-quality reference images for supervised learning or heavily depend on source images for unsupervised learning, making the fusion results challenging to accurately reflect real-world illumination conditions. To overcome the impact of unreliable factors in references, we propose a self-adaptive mean teacher-based semi-supervised learning framework tailored for MEF, termed SAMT-MEF. Its self-adaptiveness is reflected from two perspectives. Firstly, we establish a self-adaptive set to retain the best-ever outputs from the teacher as pseudo labels, employing a well-crafted hybrid metric for its updates. Secondly, we employ contrastive learning to assist the self-adaptive set further in alleviating overfitting to inferior pseudo labels. Our proposed method, backed by abundant empirical evidence, outperforms state-of-the-art methods quantitatively and qualitatively on both reference and non-reference datasets. Furthermore, in some scenarios, the fusion results surpass the reference images, showcasing superior performance in practical applications. Source code are publicly available at https://github.com/hqj9994ever/SAMT-MEF.

Fusion of infrared and visible images via multi-layer convolutional sparse representation

Infrared and visible image fusion is an effective solution for image quality enhancement. However, conventional fusion models require the decomposition of source images into image blocks, which disrupts the original structure of the images, leading to the loss of detail in the fused images and making the fusion results highly sensitive to matching errors. This paper employs Convolutional Sparse Representation (CSR) to perform global feature transformation on the source images, overcoming the drawbacks of traditional fusion models that rely on image decomposition. Inspired by neural networks, a multi-layer CSR model is proposed, which involves five layers in a forward-feeding manner: two CSR layers acquiring sparse coefficient maps, one fusion layer combining sparse maps, and two reconstruction layers for image recovery. The dataset used in this paper comprises infrared and visible images selected from public dataset, as well as registered images collected by an actual Unmanned Aerial Vehicle (UAV). The source images contain ground targets, marine targets, and natural landscapes. To validate the effectiveness of the proposed image fusion model in this paper, comparative analysis is conducted with state-of-the-art (SOTA) algorithms. Experimental results demonstrate that the proposed fusion model outperforms other state-of-the-art methods by at least 10% in SF, EN, MI and QAB/F fusion metrics in most image fusion cases, thereby affirming its favorable performance.

A Variable-Gain Fixed-Time Convergent and Robust ZNN Model for Image Fusion: Design, Analysis, and Verification

A Variable-Gain Fixed-Time Convergent and Robust ZNN Model for Image Fusion: Design, Analysis, and Verification

The diagnostic value of multimodal imaging based on MR combined with ultrasound in benign and malignant breast diseases

We aimed to construct and validate a multimodality MRI combined with ultrasound based on radiomics for the evaluation of benign and malignant breast diseases. The preoperative enhanced MRI and ultrasound images of 131 patients with breast diseases confirmed by pathology in Aerospace Center Hospital from January 2021 to August 2023 were retrospectively analyzed, including 73 benign diseases and 58 malignant diseases. Ultrasound and 3.0 T multiparameter MRI scans were performed in all patients. Then, all the data were divided into training set and validation set in a 7:3 ratio. Regions of interest were drawn layer by layer based on ultrasound and MR enhanced sequences to extract radiomics features. The optimal radiomic features were selected by the best feature screening method. Logistic Regression classifier was used to establish models according to the best features, including ultrasound model, MRI model, ultrasound combined with MRI model. The model efficacy was evaluated by the area under the curve (AUC) of the receiver operating characteristic, sensitivity, specificity, and accuracy. The F-test based on ANOVA was used to screen out 20 best ultrasonic features, 11 best MR Features, and 14 best features from the combined model. Among them, texture features accounted for the largest proportion, accounting for 79%.The ultrasound combined with MR Image fusion model based on logistic regression classifier had the best diagnostic performance. The AUC of the training group and the validation group were 0.92 and 091, the sensitivity was 0.80 and 0.67, the specificity was 0.90 and 0.94, and the accuracy was 0.84 and 0.79, respectively. It was better than the simple ultrasound model (AUC of validation set was 0.82) or the simple MR model (AUC of validation set was 0.85). Compared with the traditional ultrasound or magnetic resonance diagnosis of breast diseases, the multimodal model of MRI combined with ultrasound based on radiomics can more accurately predict the benign and malignant breast diseases, thus providing a better basis for clinical diagnosis and treatment.

Privacy‐protected object detection through trustworthy image fusion

AbstractThe neural network‐based technologies have emerged as a potent method for image fusion, object detection, and other computer vision tasks as the rapid development of deep learning. Multi‐band infrared images, in particular, capture a more extensive range of radiation details and information compared to conventional single‐band infrared images. Consequently, the fusion of multi‐band infrared images can provide more features for object detection. However, it is crucial to consider that infrared images may contain sensitive information, potentially leading to privacy concerns. Ensuring datasets privacy protection plays a crucial role in the fusion and tracking process. To address both the need for improved detection performance and the necessity for privacy protection in the infrared environment, we proposed a procedure for object detection based on multi‐band infrared image datasets and utilized the transfer learning technique to migrate knowledge learned from external infrared data to internal infrared data, thereby training the infrared image fusion model and detection model. The procedure consists of several steps: (1) data preprocessing of multi‐band infrared images, (2) multi‐band infrared image fusion, and (3) object detection. Standard evaluation metrics for image fusion and object detection ensure the authenticity of the experiments. The comprehensive validation experiments demonstrate the effectiveness of the proposed procedure in object detection tasks. Furthermore, the transfer learning can train our datasets and update the model without exposing the original data. This aspect of transfer learning is particularly beneficial for maintaining the privacy of multi‐band infrared images during the fusion and detection processes.

Deep M2CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network.

For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network.

Adversarial attacks on GAN-based image fusion

Image fusion has achieved significant success, owing to the rapid development of digital computing and Generative Adversarial Networks (GANs). GAN-based fusion techniques fuse latent codes through spatial or arithmetic operations to achieve real image fusion, facilitated by encoders. However, security concerns have arisen due to the vulnerability of deep neural networks to adversarial perturbations. This vulnerability extends to the GAN-based image fusion task. In this paper, we introduce two methods for creating adversarial examples in the context of GAN-based image fusion: adding subtle perturbations to input images or applying the adversarial patch to the input images. The subtle perturbation is meticulously crafted to be imperceptible yet capable of misleading a specific output, regardless of the input images. Conversely, the adversarial patch is a universal perturbation that, when applied to input images as a patch, induces meaningless output images. Our comprehensive experiments, conducted on datasets such as FFHQ (3 × 1024 × 1024) and Stanford Cars (3 × 512 × 512), include both quality and quantity evaluations. According to our experimental results, the subtle perturbation results in almost identical output images, while the adversarial patch induces meaningless fused images and can be transferred to other datasets. By demonstrating that image fusion models are highly vulnerable to adversarial attacks, this study highlights serious concerns regarding the security of these models.