Visible Image Fusion Research Articles

FCLFusion: A frequency-aware and collaborative learning for infrared and visible image fusion

Infrared and visible image fusion (IVIF) aims to integrate the advantages of different modal images. Most existing deep learning-based methods often focus on a single IVIF task and ignore the effect of frequency information on the fusion results, which do not fully preserve saliency structures and important texture details. The core idea of this paper is based on the following observation: (1) image content can be characterized by different frequency domain components, low frequency represents base information, such as saliency structure, while high frequency contains texture detail information. (2) multi-tasks learning generally achieve better performance than single-task. Based on these observations, we propose a fusion model called Frequency-aware and Collaborative Learning (FCLFusion) for infrared and visible images. This model takes image fusion as the main task and introduces image reconstruction as an auxiliary task to collaboratively optimize the network, thereby improving the fusion quality. Specifically, we transform spatial domain features to the frequency domain and develop a frequency feature fusion module for guiding the primary network to generate the fused image. The sub-network generates the reconstructed images. Also, we preserve the saliency and detail features via frequency skip connections. Moreover, we propose a hybrid loss function that consists of two terms: frequency loss and self-supervised reconstruction loss. The former aims to prevent information loss in the frequency domain, while the latter improves the extraction of vital information. Extensive experiments verified on three public datasets demonstrate that our FCLFusion outperforms ten state-of-the-art fusion models.

Research on fractional wavelet transform combined with parameter adaptive PCNN for infrared and visible image fusion algorithm

Infrared and visible image fusion holds significant value across various fields due to its ability to provide complementary information. However, existing fusion algorithms often suffer from susceptibility to external physical conditions of source images, leading to inconsistent fusion quality and limited adaptive capability. In this paper, we propose a fractional wavelet transform fusion algorithm for infrared and visible images. This algorithm uses fractional wavelet domain image decomposition to more accurately locate the image structures of different scales. Initially, we perform image decomposition using a non-subsampled shearlet transform (NSST) with multi-scale and multi-directional decomposition. Subsequently, to effectively fuse detailed information, we employ discrete fractional wavelet transform (DFRWT) to decompose low-frequency subbands while selecting an appropriate p-value. High-frequency subbands are fused using a parameterized adaptive pulse-coupled neural network (PCNN) model. To validate the superiority of our algorithm, we conduct visual and quantitative comparative analyses against several existing algorithms. The results confirm that our proposed algorithm exhibits robustness against external physical conditions and surpasses these existing algorithms, making it highly applicable for infrared–visible fusion tasks.

Infrared and visible image fusion based on semi-global weighted least squares and guided edge-aware filters

Most of the existing image fusion methods are dedicated to extracting the respective private features in the source image to reconstruct to get a fused image that contains rich information. However, its neglect of the enhancement of texture details in the source image as well as the preservation of background structures, which makes the fusion result look less impressive. To this end, an infrared and visible image fusion method based on a semi-global weighted least squares (SGWLS) method and guided edge-aware filter (GEAF) is proposed. First, a SGWLS method is applied to decompose the source image into base and detail layers containing different scale information. Then, the detail layer are texture-enhanced by using the scale coefficients. Due to the lack of edge features in the original base layer, for this reason, a guided edge-aware filter is designed, in which the enhanced detail layer was used as the guidance image as a way to perform a quadratic feature enhancement fusion for the base layer in order to retain richer gradient information. Finally, reconstruction of the fused detail and base layers by using inverse transformation to obtain the fusion result. Compared with nine state-of-the-art fusion algorithms, a large number of experimental results demonstrate the superior performance of the proposed method in highlighting texture details and background features, as well as the obvious advantages in SF and AG metrics.

WaveFusionNet: Infrared and visible image fusion based on multi-scale feature encoder–decoder and discrete wavelet decomposition

To merge complementary information from multimodal images, such as thermal saliency from infrared images and texture details from visible images, traditional multi-scale transform-based methods have been extensively studied, with deep learning-based methods gaining significant popularity in recent years. However, there has been limited research on optimally combining the advantages of these two categories in fusion. In this paper, we propose a novel infrared and visible image fusion (IVIF) framework, WaveFusionNet, which integrates precise frequency feature decomposition from the discrete wavelet transform (DWT) with the comprehensive feature extraction from the multi-scale encoder. Firstly, we train an encoder–decoder network for multi-scale feature extraction and image reconstruction. DWT is used for down-sampling with minimal information loss by decomposing extracted features into low and high-frequency sub-bands. Next, a dual-band feature fusion (DBFF) module is trained to merge these sub-bands by integrating a spatial feature transform-based sub-network for low-frequency fusion and a maximum absolute value selection strategy for fusing high-frequencies. Finally, all fused sub-bands are fed into the pre-trained decoder to reconstruct the final image. Experimental results on three benchmark datasets (TNO, Roadscene, and MSRS) demonstrate that the proposed fusion method outperforms recent IVIF methods in both quantitative assessment and visual perception while maintaining competitive time complexity.

MBHFuse: A multi- branch heterogeneous global and local infrared and visible image fusion with differential convolutional amplification features

Fusing infrared and visible imagery aims to harness their complementary spectral data, enhancing output image quality, sharpness, and content. However, convolutional neural networks (CNNs) are not capable of capturing long-range dependencies, while the Transformer architecture faces the challenge of consuming huge computational resources. To address these critical challenges, this paper proposes a novel framework named MBHFuse, which employs a multi-branch heterogeneous global and local image fusion approach. To effectively extract global and local features, we design a multi-branch heterogeneous encoder module and introduce a differential convolution amplification module (DCAM) to further extract complementary information. Additionally, we devise a new loss function, incorporating multi-branch feature decomposition loss, intensity loss, gradient loss, mean squared error loss, and structural similarity loss, for training the proposed MBHFuse model. Through extensive experiments on public datasets, we demonstrate that the proposed framework outperforms other state-of-the-art (SOTA) methods in both qualitative and quantitative evaluations. Our code will be available at https://github.com/sunyichen1994/MBHFuse.

BCMFIFuse: A Bilateral Cross-Modal Feature Interaction-Based Network for Infrared and Visible Image Fusion

The main purpose of infrared and visible image fusion is to produce a fusion image that incorporates less redundant information while incorporating more complementary information, thereby facilitating subsequent high-level visual tasks. However, obtaining complementary information from different modalities of images is a challenge. Existing fusion methods often consider only relevance and neglect the complementarity of different modalities’ features, leading to the loss of some cross-modal complementary information. To enhance complementary information, it is believed that more comprehensive cross-modal interactions should be provided. Therefore, a fusion network for infrared and visible fusion is proposed, which is based on bilateral cross-feature interaction, termed BCMFIFuse. To obtain features in images of different modalities, we devise a two-stream network. During the feature extraction, a cross-modal feature correction block (CMFC) is introduced, which calibrates the current modality features by leveraging feature correlations from different modalities in both spatial and channel dimensions. Then, a feature fusion block (FFB) is employed to effectively integrate cross-modal information. The FFB aims to explore and integrate the most discriminative features from the infrared and visible image, enabling long-range contextual interactions to enhance global cross-modal features. In addition, to extract more comprehensive multi-scale features, we develop a hybrid pyramid dilated convolution block (HPDCB). Comprehensive experiments on different datasets reveal that our method performs excellently in qualitative, quantitative, and object detection evaluations.

CsdlFusion: An Infrared and Visible Image Fusion Method Based on LatLRR-NSST and Compensated Saliency Detection

CsdlFusion: An Infrared and Visible Image Fusion Method Based on LatLRR-NSST and Compensated Saliency Detection

MSCS: Multi-stage feature learning with channel-spatial attention mechanism for infrared and visible image fusion

The intention of infrared and visible image fusion is to combine the images captured by different modal sensors in the same scene to enhance its understanding. Deep learning has been proven its powerful application in image fusion due to its fine generalization, robustness, and representability of deep features. However, the performance of these deep learning-based methods heavily depends on the illumination condition. Especially in dark or exposed scenes, the fused results are over-smoothness and low-contrast, resulting in inaccuracy of object detection. To address these issues, this paper develops a multi-stage feature learning approach with channel-spatial attention mechanism, namely MSCS, for infrared and visible image fusion. The MSCS is composed of the following four key procedures: Firstly, the infrared and visible images are decomposed into illumination and reflectance components by a proposed network called as Retinex_Net. Then, the components are transported to an encoder for features coding. Next, we propose an adaptive fusion module with attention mechanisms to fuse the features. Finally, the fused image is generated by the decoder for decoding the fused features. Meanwhile, a novel fusion loss function and a multi-stage training strategy are proposed to train the above modules. The subjective and objective results of experiments on TNO, LLVIP and MSRS datasets illustrate that the proposed method is effective and performs better than the state-of-the-art fusion methods on achieving enjoyable results in dark or over-exposure scenes. And the results of further experiments on the fused images for object detection demonstrate that the fusion outputs produced by our MSCS are more beneficial for detection tasks.

MADMFuse: A multi-attribute diffusion model to fuse infrared and visible images

In the field of deep learning vision, infrared and visible image fusion (IVIF) has received significant attention due to its ability to enhance scene comprehension by combining the complementary features of both types of images. Existing methods based on generative models are either unstable in training, such as generative adversarial network, or only have a single denoising object, such as diffusion models, resulting in poor fused results that cannot fully contain multi-modal features. To solve these problems, we propose a multi-attribute diffusion model to fuse infrared and visible images, termed MADMFuse. Specifically, to preserve the salient information in infrared images and the texture details in visible images, we have designed a forward diffusion process with shared noise for multiple independent attributes, ensuring that the denoising network learns complementary features of different attributes simultaneously during training. Subsequently, our reverse process with multi-attribute as conditions employs the denoising network that iteratively reparameterizes noise, gradually adjusting and achieving fine image fusion. Furthermore, to address the issue of feature degradation resulting from solely focusing on a single denoising object, we derive a multiple diffusion object loss function based on pixel-level fusion target, the fidelity and luminance term in this loss function aim to guide the fused results towards visual similarity with the source image while preserving its brightness. Extensive experiments indicate that MADMFuse is more effective than other state-of-the-art image fusion methods.

MFTCFNet: infrared and visible image fusion network based on multi-layer feature tightly coupled

MFTCFNet: infrared and visible image fusion network based on multi-layer feature tightly coupled

Enhancing three-source cross-modality image fusion with improved DenseNet for infrared polarization and visible light images

The fusion of multi-modal images to create an image that preserves the unique features of each modality as well as the features shared across modalities is a challenging task, particularly in the context of infrared (IR)-visible image fusion. In addition, the presence of polarization and IR radiation information in images obtained from IR polarization sensors further complicates the multi-modal image-fusion process. This study proposes a fusion network designed to overcome the challenges associated with the integration of low-resolution IR, IR polarization, and high-resolution visible (VIS) images. By introducing cross attention modules and a multi-stage fusion approach, the network can effectively extract and fuse features from different modalities, fully expressing the diversity of the images. This network learns end-to-end mapping from sourced to fused images using a loss function, eliminating the need for ground-truth images for fusion. Experimental results using public datasets and remote-sensing field-test data demonstrate that the proposed methodology achieves commendable results in qualitative and quantitative evaluations, with gradient based fusion performance QAB/F, mutual information (MI), and QCB values higher than the second-best values by 0.20, 0.94, and 0.04, respectively. This study provides a comprehensive representation of target scene information that results in enhanced image quality and improved object identification capabilities. In addition, outdoor and VIS image datasets are produced, providing a data foundation and reference for future research in related fields.

Edge-oriented unrolling network for infrared and visible image fusion

Edge-oriented unrolling network for infrared and visible image fusion

Soft computing-driven infrared and visible image fusion network for security application service

The high-level visual tasks (HLVT)-specific infrared and visible image fusion networks aim to guide the fusion network to focus on specific feature regions. The resulting fused images enhance the performance of HLVT in adverse conditions. However, cross-layer visual task fusion may diminish the fusion network’s adaptability to diverse data. This deficiency results in a lack of robustness and security in the fusion network. Consequently, the fusion network is rendered insufficient to fulfill the requirements of HLVT security applications, including object detection and security systems. To address these challenges, we propose a soft computing-driven infrared and visible image fusion network based on layer separation and dual-stream adversarial learning, referred to as LsDSGFuse. Firstly, we employ a layer segmentation network based on a high-resolution network to create salient object segmentation masks for the original image. This helps mitigate the impact of task-specific guidance on the fusion network learning process, preventing overfitting to HLVT. Secondly, we use the original image after information separation as the input of the generator. The separated network inputs clearly define foreground objects and background environments, aiding the generator in better fusing foreground and background. Lastly, we adopt a no-loading training strategy to balance adversarial learning, prevent discriminator overfitting, and reduce resource waste. Experimental results demonstrate the outstanding performance of our proposed image fusion method in terms of fusion quality, surpassing several representative approaches. Furthermore, experiments in HLVT indicate that our method facilitates HLVT in better responding to potential threats and unknown variables. This improvement significantly enhances HLVT’s security performance.

Infrared and Harsh Light Visible Image Fusion Using an Environmental Light Perception Network.

The complementary combination of emphasizing target objects in infrared images and rich texture details in visible images can effectively enhance the information entropy of fused images, thereby providing substantial assistance for downstream composite high-level vision tasks, such as nighttime vehicle intelligent driving. However, mainstream fusion algorithms lack specific research on the contradiction between the low information entropy and high pixel intensity of visible images under harsh light nighttime road environments. As a result, fusion algorithms that perform well in normal conditions can only produce low information entropy fusion images similar to the information distribution of visible images under harsh light interference. In response to these problems, we designed an image fusion network resilient to harsh light environment interference, incorporating entropy and information theory principles to enhance robustness and information retention. Specifically, an edge feature extraction module was designed to extract key edge features of salient targets to optimize fusion information entropy. Additionally, a harsh light environment aware (HLEA) module was proposed to avoid the decrease in fusion image quality caused by the contradiction between low information entropy and high pixel intensity based on the information distribution characteristics of harsh light visible images. Finally, an edge-guided hierarchical fusion (EGHF) module was designed to achieve robust feature fusion, minimizing irrelevant noise entropy and maximizing useful information entropy. Extensive experiments demonstrate that, compared to other advanced algorithms, the method proposed fusion results contain more useful information and have significant advantages in high-level vision tasks under harsh nighttime lighting conditions.

A three-dimensional feature-based fusion strategy for infrared and visible image fusion

Due to the lacking of attention to the scene’s essential characteristics, the existing fusion methods suffer from the deficiency of scene distortion. In addition, the lack of groundtruth can cause an inadequate representation of vital information. To this end, we propose a novel infrared and visible image fusion network based on three-dimensional feature fusion strategy (D3Fuse). In our method, we consider the scene semantic information in the source images and extract the commonality contents of the two images as the third-dimensional feature to extend the feature space for fusion tasks. Specifically, a commonality feature extraction module (CFEM) is designed to extract the scene commonality features. Subsequently, the scene commonality features are utilized together with modality features to construct the fusion image. Moreover, to ensure the independence and diversity of distinct features, we employ a contrastive learning strategy with multiscale PCA coding, which stretches the feature distance in an unsupervised manner, prompting the encoder to extract more discriminative information without incurring additional parameters and computational costs. Furthermore, a contrastive enhancement strategy is utilized to ensure adequate representation of modality information. The results of the qualitative and quantitative evaluations on the three datasets show that the proposed method has better visual performance and higher objective metrics with lower computational cost. The object detection experiments show that our results have superior performance on high-level semantic tasks.

IMQFusion: Infrared and visible image fusion via implicit multi-resolution preservation and query aggregation

The infrared and visible image fusion (IVIF) task aims to generate high-resolution images with richer texture details and finer salient information to enhance visual understanding for downstream tasks. However, current IVIF techniques often focus on exploring complementary information from source images at a single resolution, neglecting valuable information available at other resolutions. To effectively exploit multi-resolution semantics, we propose a novel framework called IMQFusion, which enhances fusion performance by leveraging intra-modality multi-resolution preservation and inter-modality multi-resolution query aggregation. Specifically, our method introduces an implicit neural representation (INR) with a robust continuous map that vertically reduces the semantic gap among adjacent paired resolutions, avoiding reliance on simple upsampling. Additionally, we employ a bidirectional guided strategy to horizontally facilitate resolution preservation. Moreover, to integrate multi-resolution semantics across both modalities and resolutions effectively, we develop an efficient aggregation strategy that incorporates a query mechanism, thereby enhancing inter-modality interactions among multi-resolution features. Extensive experiments demonstrate that, compared to state-of-the-art IVIF methods, the fused images generated by IMQFusion exhibit superior visual balance between salient information and texture details, as well as higher quantitative metrics. Furthermore, IMQFusion possesses the potential to improve object detection and can be extended to the field of medical image fusion.

Siam-AUnet: An end-to-end infrared and visible image fusion network based on gray histogram

Infrared and visible image fusion aim to leverage the respective advantages of both modalities to extract richer information from a scene. In environments with ample lighting, it is imperative to preserve the characteristics of visible images and salvage details lost due to overexposure. Conversely, in dark environments, a preference is typically inclined towards the utilization of infrared imagery. However, current methodologies are unable to adaptively modulate their tendency towards either modality, thereby resulting in suboptimal fusion quality. A gray histogram visually represents the dynamic range of image distribution, revealing the contrast and degree of brightness variations. Drawing from this insight, in this paper, we propose a novel image fusion method called Siam-AUnet for the first time. This methodology transforms the distribution of luminance histograms extracted from visible images into weights during the fusion of the two modalities. We adopt a Siamese network architecture and a multi-scale attention mechanism to focus more effectively on key features while reducing the number of parameters. Extensive experiments on different datasets and comparisons with other methods have demonstrated the superiority of this fusion approach. By utilizing the characteristics of the histogram, Siam-AUnet can adaptively tend towards the modality with clearer texture details. Qualitative and quantitative experimental results on the MSRS test dataset demonstrate that Siam-AUnet achieves optimal performance in terms of EN, SSIM, and MS_SSIM, while exhibiting suboptimal performance in metrics such as SCD, VIF, and Qqbf. In well-lit conditions, it can preserve the detailed features of visible images and the salient features of infrared images; while under adverse lighting conditions, it emphasizes the infrared modality features more strongly. Our code will be publicly available at https://github.com/xkangKK/Siam-AUnet.

An infrared and visible image fusion using knowledge measures for intuitionistic fuzzy sets and Swin Transformer

The objectives of infrared and visible image fusion are to generate a single image that includes significant objects and rich texture information. However, the current deep-learning methods ignore uncertainty in the decision-making process during the fusion phase. To address this issue, we propose a novel infrared and visible image fusion method using a Swin Transformer and knowledge measures of intuitionistic fuzzy sets (IFSs) named SWKIF-Fusion. This model employs a Swin Transformer-based pre-trained module for feature extraction, which is the most effective module for modeling long-range dependencies. The fusion process of SWKIF-Fusion integrates the proposed knowledge measure of IFSs. IFSs inherently possess a high capability to handle uncertainty, and the knowledge measure of IFSs provides uncertainty quantification. This integration in the fusion phase mitigates the uncertainty in the decision-making process. This fusion of IFSs, knowledge measures, and the Swin Transformer-based deep learning model enhances the overall performance, as demonstrated through experiments on the TNO, Roadscene, OTCBVS, M3FD, and MSRS datasets. This study also fills the gap in developing knowledge measures for IFSs by proposing novel general construction methods. It presents a theoretically sound framework for knowledge measures of IFSs using well-established mathematical concepts such as t-norms, t-conorms, automorphisms, and aggregation operators.

CMFuse: Cross-Modal Features Mixing via Convolution and MLP for Infrared and Visible Image Fusion

CMFuse: Cross-Modal Features Mixing via Convolution and MLP for Infrared and Visible Image Fusion

DGFusion: An effective dynamic generalizable network for infrared and visible image fusion

The objective of infrared and visible image fusion is to generate a unified image that highlights prominent targets and retains intricate texture details, even in scenarios with imbalanced source image information. However, current image fusion algorithms primarily consider factors like illumination, restricting their applicability to certain scenes and compromising their adaptability. To tackle the issue, this paper proposes the DGFusion, which utilizes TWSSLoss to balance the contribution of source images in the fused output, effectively mitigating the limitations linked to relying solely on illumination guidance. Additionally, modality-complement feature attention harmonizer (MCFAH) facilitates cross-modal channel attention learning. This process assigns weights to features and accomplishes fusion by exchanging cross-modal differential information, thereby enriching each feature with details from the other modality. Furthermore, the multi convolution attentive net (MCAN) dynamically adjusts the contributions of features from different modalities. It prioritizes the most expressive characteristics to accentuate complementary information, enabling efficient fusion. In conclusion, our method outperforms seven state-of-the-art alternatives in terms of preserving target details and retaining texture information. Rigorous generalization experiments across five diverse datasets demonstrate the robustness of our DGFusion model in various scenarios.