Visible Image Fusion Research Articles

A multi-scale information integration framework for infrared and visible image fusion

Infrared and visible image fusion aims at generating a fused image containing the intensity and detail information of source images, and the key issue is effectively measuring and integrating the complementary information of multi-modality images from the same scene. Existing methods mostly adopt a simple weight in the loss function to decide the information retention of each modality rather than adaptively measuring complementary information for different image pairs. In this study, we propose a multi-scale dual attention (MDA) framework for infrared and visible image fusion, which is designed to measure and integrate multi-scale complementary information in both structure and loss function at the image and patch level. In our method, the residual downsample block decomposes source images into three scales first. Then, dual attention fusion block integrates complementary information and generates a spatial and channel attention map at the same and adjacent scale for feature fusion. Finally, the output image is reconstructed by the residual reconstruction block. Loss function consists of image-level, feature-level and patch-level three parts, of which the calculation of the image-level and patch-level two parts are based on the weights generated by the complementary information measurement. Indeed, to constrain the pixel intensity distribution between the output and infrared image, a style loss is added. Our fusion results perform robust and informative across different scenarios. Qualitative and quantitative results on three datasets illustrate that our method is able to preserve both thermal radiation and detailed information from two modalities and achieve comparable results compared with the other state-of-the-art methods. Ablation experiments show the effectiveness of our information integration architecture and adaptively measure complementary information retention in the loss function. Our code is available at https://github.com/SSyangguang/MDA.

Infrared and visible image fusion and detection based on interactive training strategy and feature filter extraction module

The infrared and visible image fusion-detection technique aims to expand the observation wavelength range to overcome harsh environmental conditions such as low light, smoke, heavy fog, etc., and improve the reliability and robustness of target detection. However, the current fusion-detection method suffers from problems of the joint optimization of the fusion-detection model and irrelevant feature interference. To address these problems, we propose an infrared and visible image fusion-detection network based on an interactive training strategy (ITS) and a feature filter extraction module (FFEM). The ITS constructs the closed-loop optimization for the fusion-detection method, which can mutually enhance the performance of the fusion and detection model through mutual supervision. The FFEM is designed to acquire detection-conducive features from visible and infrared images. Specifically, we define filtering weights based on the covariance values of features and use these weights to enhance relevant features and soft redundant and irrelevant features (e.g., noise). Quantitative and qualitative experimental results demonstrate that the proposed method achieves competitive performance in target detection accuracy compared with other state-of-the-art fusion-detection methods.

Semantic attention-based heterogeneous feature aggregation network for image fusion

Infrared and visible image fusion aims to generate a comprehensive image that retains both salient targets of the infrared image and texture details of the visible image. However, existing methods overlook the differences in attention to semantic information among different fused images. To address this issue, we propose a semantic attention-based heterogeneous feature aggregation network for image fusion. The key component of our network is the semantic attention-based fusion module, which leverages the weights derived from semantic feature maps to dynamically adjust the significance of various semantic objects within the fusion feature maps. By using semantic weights as guidance, our fusion process concentrates on regions with crucial semantics, resulting in a more focused fusion that preserves rich semantic information. Moreover, we propose an innovative component called the attentive dense block. This block effectively filters out irrelevant features during extraction, accentuates essential features to their maximum potential, and enhances the visual quality of the fused images. Importantly, our network demonstrates strong generalization capabilities. Extensive experiments validate the superiority of our proposed network over current state-of-the-art techniques in terms of both visual appeal and semantics-driven evaluation.

A self-supervised fusion for infrared and visible images via multi-level contrastive auto-encoding

The objective of infrared and visible image fusion is to integrate the features of source images and acquire a fused image that highlights foreground objects and preserves background details. Nonetheless, due to ignoring the mutual information interaction between source images, prevailing image fusion methods always encounter challenges related to incomplete or redundant feature extraction. For this purpose, this study presents a self-supervised fusion framework,employing multi-level contrastive auto-encoding to realize the infrared and visible image fusion (IVIF) task, referred to as SS-MCAE. The SS-MCAE consists of a coarse-to-fine feature extraction network (CFEN) and a fusion network (FN), where CFEN contains a content-aware enhancement module (CEM) and a multi-level contrastive learning autoencoder (MCLA). Specifically, CFEN employs CEM to adaptively enhance brightness information and texture details from source images and the MCLA is introduced to refine and extract these features hierarchically. Within contrastive learning, a neoteric sample generator (SG) is introduced for constructing positive and negative samples, aiming to maximize feature extraction while diminishing redundant information. Furthermore, an innovative perceptual loss is developed to retain original features and guide the image reconstruction, establishing a more reliable relationship between extracted features and source images. Comprehensive experiments reveal that our proposed SS-MCAE is superior to current approaches in both visual effect and quantitative analysis.

Infrared and visible image fusion based on FUDPCNN and gravitational force operator

Infrared and visible image fusion based on FUDPCNN and gravitational force operator

Tongue image fusion and analysis of thermal and visible images in diabetes mellitus using machine learning techniques

The study aimed to achieve the following objectives: (1) to perform the fusion of thermal and visible tongue images with various fusion rules of discrete wavelet transform (DWT) to classify diabetes and normal subjects; (2) to obtain the statistical features in the required region of interest from the tongue image before and after fusion; (3) to distinguish the healthy and diabetes using fused tongue images based on deep and machine learning algorithms. The study participants comprised of 80 normal subjects and age- and sex-matched 80 diabetes patients. The biochemical tests such as fasting glucose, postprandial, Hba1c are taken for all the participants. The visible and thermal tongue images are acquired using digital single lens reference camera and thermal infrared cameras, respectively. The digital and thermal tongue images are fused based on the wavelet transform method. Then Gray level co-occurrence matrix features are extracted individually from the visible, thermal, and fused tongue images. The machine learning classifiers and deep learning networks such as VGG16 and ResNet50 was used to classify the normal and diabetes mellitus. Image quality metrics are implemented to compare the classifiers’ performance before and after fusion. Support vector machine outperformed the machine learning classifiers, well after fusion with an accuracy of 88.12% compared to before the fusion process (Thermal-84.37%; Visible-63.1%). VGG16 produced the classification accuracy of 94.37% after fusion and attained 90.62% and 85% before fusion of individual thermal and visible tongue images, respectively. Therefore, this study results indicates that fused tongue images might be used as a non-contact elemental tool for pre-screening type II diabetes mellitus.

DCFNet: Infrared and Visible Image Fusion Network Based on Discrete Wavelet Transform and Convolutional Neural Network.

Aiming to address the issues of missing detailed information, the blurring of significant target information, and poor visual effects in current image fusion algorithms, this paper proposes an infrared and visible-light image fusion algorithm based on discrete wavelet transform and convolutional neural networks. Our backbone network is an autoencoder. A DWT layer is embedded in the encoder to optimize frequency-domain feature extraction and prevent information loss, and a bottleneck residual block and a coordinate attention mechanism are introduced to enhance the ability to capture and characterize the low- and high-frequency feature information; an IDWT layer is embedded in the decoder to achieve the feature reconstruction of the fused frequencies; the fusion strategy adopts the l1-norm fusion strategy to integrate the encoder's output frequency mapping features; a weighted loss containing pixel loss, gradient loss, and structural loss is constructed for optimizing network training. DWT decomposes the image into sub-bands at different scales, including low-frequency sub-bands and high-frequency sub-bands. The low-frequency sub-bands contain the structural information of the image, which corresponds to the important target information, while the high-frequency sub-bands contain the detail information, such as edge and texture information. Through IDWT, the low-frequency sub-bands that contain important target information are synthesized with the high-frequency sub-bands that enhance the details, ensuring that the important target information and texture details are clearly visible in the reconstructed image. The whole process is able to reconstruct the information of different frequency sub-bands back into the image non-destructively, so that the fused image appears natural and harmonious visually. Experimental results on public datasets show that the fusion algorithm performs well according to both subjective and objective evaluation criteria and that the fused image is clearer and contains more scene information, which verifies the effectiveness of the algorithm, and the results of the generalization experiments also show that our network has good generalization ability.

SCGAFusion: A skip-connecting group convolutional attention network for infrared and visible image fusion

The goal of infrared and visible image fusion is to synthesize an image that contains complementary information as much as possible in paired infrared and visible images. However, due to using the convolution operation with local receptive field to extract features, most of the existing deep learning-based fusion methods fail to effectively exploit global contextual information in source images and so obtain a limited fusion performance. To address this issue, this paper proposes an end-to-end skip-connecting group convolutional attention network, termed as SCGAFusion, to fuse infrared and visible images. In SCGAFusion, a group convolutional attention block (GAB) is purposefully developed to promote the extraction and utilization capability of the fusion network for informative features. GAB introduces group convolutional layer to obtain hierarchical features, and employs a residual non-local attention module to capture long-range dependencies between pixels. In this way, it can not only focus more on the important regions or details, but also leverage both local neighborhood and global contextual information. Additionally, a hierarchical feature compensation mechanism based on skip connections is devised to integratedly exploit both local and global features. Experimental results on several public datasets qualitatively and quantitatively demonstrate the advantage of SCGAFusion over other state-of-the-art fusion methods.

Infrared and Visible Image Fusion Algorithm Based on Double-Domain Transform Filter and Contrast Transform Feature Extraction

Current challenges in visible and infrared image fusion include color information distortion, texture detail loss, and target edge blur. To address these issues, a fusion algorithm based on double-domain transform filter and nonlinear contrast transform feature extraction (DDCTFuse) is proposed. First, for the problem of incomplete detail extraction that exists in the traditional transform domain image decomposition, an adaptive high-pass filter is proposed to decompose images into high-frequency and low-frequency portions. Second, in order to address the issue of fuzzy fusion target caused by contrast loss during the fusion process, a novel feature extraction algorithm is devised based on a novel nonlinear transform function. Finally, the fusion results are optimized and color-corrected by our proposed spatial-domain logical filter, in order to solve the color loss and edge blur generated in the fusion process. To validate the benefits of the proposed algorithm, nine classical algorithms are compared on the LLVIP, MSRS, INO, and Roadscene datasets. The results of these experiments indicate that the proposed fusion algorithm exhibits distinct targets, provides comprehensive scene information, and offers significant image contrast.

A Visible and Synthetic Aperture Radar Image Fusion Algorithm Based on a Transformer and a Convolutional Neural Network

For visible and Synthetic Aperture Radar (SAR) image fusion, this paper proposes a visible and SAR image fusion algorithm (TCFuse) based on a Transformer and a Convolutional Neural Network (CNN). Firstly, in this paper, the Restormer Block is used to extract cross-modal shallow features. Then, we introduce an improved Transformer–CNN Feature Extractor (TCFE) with a two-branch residual structure. This includes a Transformer branch that introduces the Lite Transformer (LT) and DropKey for extracting global features and a CNN branch that introduces the Convolutional Block Attention Module (CBAM) for extracting local features. Finally, the fused image is output based on global features extracted by the Transformer branch and local features extracted by the CNN branch. The experiments show that the algorithm proposed in this paper can effectively achieve the extraction and fusion of global and local features of visible and SAR images, so that high-quality visible and SAR fusion images can be obtained.

Fidelity based visual compensation and salient information rectification for infrared and visible image fusion

The fusion technology, combining infrared and visible modes, has the potential to enhance the semantic content of backgrounds, thereby improving scene interpretability. However, most existing image fusion algorithms primarily concentrate on the fusion process, often neglecting the importance of preprocessing the source images to enhance their visual fidelity. Additionally, these algorithms frequently overlook the distinct characteristics of infrared and visible modes, leading to suboptimal weight allocations that do not correspond with human perception. To tackle these issues, this paper proposes a fusion algorithm that emphasizes visual fidelity and the rectification of salient information. More specifically, we improve fusion algorithms by designing an adaptive enhancement method based on Taylor approximation and visual compensation, which proves particularly effective in complex environments. Our proposed multi-scale decomposition approach extracts salient information from the transmission map, thereby enriching fusion results with finer details to accentuate target features. Drawing inspiration from the distinctive attributes of infrared and visible image modes, we devise a fusion weight calculation method grounded in similarity measurements to effectively convey significant information from the source images. To validate the effectiveness of our proposed method, we conducted validation experiments using publicly available datasets. Our experimental findings exhibit a prominent advantage over fifteen state-of-the-art fusion algorithms in both subjective and objective assessments. Our code is publicly available at: https://github.com/VCMHE/FVC_SIR.

Fusion of infrared and visible images via multi-layer convolutional sparse representation

Infrared and visible image fusion is an effective solution for image quality enhancement. However, conventional fusion models require the decomposition of source images into image blocks, which disrupts the original structure of the images, leading to the loss of detail in the fused images and making the fusion results highly sensitive to matching errors. This paper employs Convolutional Sparse Representation (CSR) to perform global feature transformation on the source images, overcoming the drawbacks of traditional fusion models that rely on image decomposition. Inspired by neural networks, a multi-layer CSR model is proposed, which involves five layers in a forward-feeding manner: two CSR layers acquiring sparse coefficient maps, one fusion layer combining sparse maps, and two reconstruction layers for image recovery. The dataset used in this paper comprises infrared and visible images selected from public dataset, as well as registered images collected by an actual Unmanned Aerial Vehicle (UAV). The source images contain ground targets, marine targets, and natural landscapes. To validate the effectiveness of the proposed image fusion model in this paper, comparative analysis is conducted with state-of-the-art (SOTA) algorithms. Experimental results demonstrate that the proposed fusion model outperforms other state-of-the-art methods by at least 10% in SF, EN, MI and QAB/F fusion metrics in most image fusion cases, thereby affirming its favorable performance.

DeDNet: Infrared and visible image fusion with noise removal by decomposition-driven network

Infrared and visible image fusion integrates useful information from different modal images to generate one with comprehensive details and highlighted targets, thereby deepening the interpretation of the scene. However, existing deep-learning based methods do not consider noise, leading to suboptimal noisy fusion results. To address this issue, we propose a decomposition-driven neural network (DeDNet) to achieve joint fusion and noise removal. By introducing constraints between the fused and ground truth source images into the loss function, we develop an autoencoder as the basic fusion and denoising network. Furthermore, we propose a decomposition network that guided the decomposition of the fusion result, improving the denoising and details recovery. Experiments demonstrate DeDNet excels the state-of-the-art methods in objective and subjective evaluations, yielding competing performance in detection and segmentation. On the metrics,Qcb, EN, SSIM, PSNR, and CC, DeDNet average increased 10.92%, 21.13%, 82.97%, 8.55%, and 16.26% than the compared methods, respectively. The source code is available at https://github.com/JasonWong30/DeDNet.

DFG-HCEN: A distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image fusion

In this paper, we propose an unsupervised learning approach for the task of infrared and visible image fusion. This approach is called a distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image Fusion (DFG-HCEN). Instead of using complex fusion rules, DFG-HCEN uses multi-level fusion to achieve fusion results, effectively avoiding information loss during feature extraction. To improve the fusion effect, we designed a distinctive-feature guided module that strengthens the relationship between modules. Moreover, the proposed hierarchical channel enhanced and distinctive-Feature guided module aims to facilitate the fusion framework in efficiently integrating the multilevel complementary features of the source pictures. In addition, we incorporate a hybrid loss method for unsupervised training of the provided DFG-HCEN. The fidelity loss is used to constrain the pixel similarity between the fused result and source images. The application of luminance regularization loss has been shown to be an efficient method for addressing the problem of luminance degradation in fused images. We conducted extensive experiments, including visual examination and quantitative analysis, comparing DFG-HCEN with thirteen other state-of-the-art fusion techniques. The results demonstrate the superiority of DFG-HCEN. Moreover, the extended object detection experiments validate the ability of DFG-HCEN to fully support downstream tasks.

Infrared and Visible Image Fusion (IVF) Using Latent Low-Rank Representation and Deep Feature Extraction Network

Infrared and Visible Image Fusion (IVF) Using Latent Low-Rank Representation and Deep Feature Extraction Network

Photovoltaic fault detection based on infrared and visible image augmentation and fusion

Photovoltaic fault detection based on infrared and visible image augmentation and fusion

Semantic-Aware Fusion Network Based on Super-Resolution.

The aim of infrared and visible image fusion is to generate a fused image that not only contains salient targets and rich texture details, but also facilitates high-level vision tasks. However, due to the hardware limitations of digital cameras and other devices, there are more low-resolution images in the existing datasets, and low-resolution images are often accompanied by the problem of losing details and structural information. At the same time, existing fusion algorithms focus too much on the visual quality of the fused images, while ignoring the requirements of high-level vision tasks. To address the above challenges, in this paper, we skillfully unite the super-resolution network, fusion network and segmentation network, and propose a super-resolution-based semantic-aware fusion network. First, we design a super-resolution network based on a multi-branch hybrid attention module (MHAM), which aims to enhance the quality and details of the source image, enabling the fusion network to integrate the features of the source image more accurately. Then, a comprehensive information extraction module (STDC) is designed in the fusion network to enhance the network's ability to extract finer-grained complementary information from the source image. Finally, the fusion network and segmentation network are jointly trained to utilize semantic loss to guide the semantic information back to the fusion network, which effectively improves the performance of the fused images on high-level vision tasks. Extensive experiments show that our method is more effective than other state-of-the-art image fusion methods. In particular, our fused images not only have excellent visual perception effects, but also help to improve the performance of high-level vision tasks.

GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

SimpliFusion: a simplified infrared and visible image fusion network

SimpliFusion: a simplified infrared and visible image fusion network

A deep learning based relative clarity classification method for infrared and visible image fusion

Infrared and visible image fused(IVIF) results normally suffer from detail loss, noise occurrence, low contrast, and blurred edges. In this paper, a new method is proposed to address the detail loss, low contrast, and blurring issue in IVIF. Specifically, visible images are enhanced by guided filter and high dynamic range compression. Infrared images are normalized by a linear transformation. Then we use blur and clear discrimination to detect salient pixels between infrared and visible images. A fully weight shared multi-path residual neural network is proposed for blur discrimination between infrared and visible image pixels in the same position. Clear pixels are treated as salient pixels, which contribute more to fused images than blur pixels. The output of our proposed network is a binary classification map for blur and clear discrimination, which is treated as our fusion weight map in the fusion stage. To deal with the discontinuous problem, we compute the two distance-transformed maps of the binary classification map and its complementary map. The two distance-transformed maps are used as weight maps to fuse the enhanced infrared and visible images. Finally, we use single scale retinex (SSR) to further enhance our fused images. The experimental results in public IVIF datasets demonstrate the superior performance of our proposed approach over other state-of-the-art methods in terms of both subjective visual quality and objective metrics. The source code is available in https://github.com/eyob12/Multi_path_residual_neural_network_based_IVIF