Visible Image Fusion Research Articles

Fidelity based visual compensation and salient information rectification for infrared and visible image fusion

The fusion technology, combining infrared and visible modes, has the potential to enhance the semantic content of backgrounds, thereby improving scene interpretability. However, most existing image fusion algorithms primarily concentrate on the fusion process, often neglecting the importance of preprocessing the source images to enhance their visual fidelity. Additionally, these algorithms frequently overlook the distinct characteristics of infrared and visible modes, leading to suboptimal weight allocations that do not correspond with human perception. To tackle these issues, this paper proposes a fusion algorithm that emphasizes visual fidelity and the rectification of salient information. More specifically, we improve fusion algorithms by designing an adaptive enhancement method based on Taylor approximation and visual compensation, which proves particularly effective in complex environments. Our proposed multi-scale decomposition approach extracts salient information from the transmission map, thereby enriching fusion results with finer details to accentuate target features. Drawing inspiration from the distinctive attributes of infrared and visible image modes, we devise a fusion weight calculation method grounded in similarity measurements to effectively convey significant information from the source images. To validate the effectiveness of our proposed method, we conducted validation experiments using publicly available datasets. Our experimental findings exhibit a prominent advantage over fifteen state-of-the-art fusion algorithms in both subjective and objective assessments. Our code is publicly available at: https://github.com/VCMHE/FVC_SIR.

Fusion of infrared and visible images via multi-layer convolutional sparse representation

Infrared and visible image fusion is an effective solution for image quality enhancement. However, conventional fusion models require the decomposition of source images into image blocks, which disrupts the original structure of the images, leading to the loss of detail in the fused images and making the fusion results highly sensitive to matching errors. This paper employs Convolutional Sparse Representation (CSR) to perform global feature transformation on the source images, overcoming the drawbacks of traditional fusion models that rely on image decomposition. Inspired by neural networks, a multi-layer CSR model is proposed, which involves five layers in a forward-feeding manner: two CSR layers acquiring sparse coefficient maps, one fusion layer combining sparse maps, and two reconstruction layers for image recovery. The dataset used in this paper comprises infrared and visible images selected from public dataset, as well as registered images collected by an actual Unmanned Aerial Vehicle (UAV). The source images contain ground targets, marine targets, and natural landscapes. To validate the effectiveness of the proposed image fusion model in this paper, comparative analysis is conducted with state-of-the-art (SOTA) algorithms. Experimental results demonstrate that the proposed fusion model outperforms other state-of-the-art methods by at least 10% in SF, EN, MI and QAB/F fusion metrics in most image fusion cases, thereby affirming its favorable performance.

DeDNet: Infrared and visible image fusion with noise removal by decomposition-driven network

Infrared and visible image fusion integrates useful information from different modal images to generate one with comprehensive details and highlighted targets, thereby deepening the interpretation of the scene. However, existing deep-learning based methods do not consider noise, leading to suboptimal noisy fusion results. To address this issue, we propose a decomposition-driven neural network (DeDNet) to achieve joint fusion and noise removal. By introducing constraints between the fused and ground truth source images into the loss function, we develop an autoencoder as the basic fusion and denoising network. Furthermore, we propose a decomposition network that guided the decomposition of the fusion result, improving the denoising and details recovery. Experiments demonstrate DeDNet excels the state-of-the-art methods in objective and subjective evaluations, yielding competing performance in detection and segmentation. On the metrics,Qcb, EN, SSIM, PSNR, and CC, DeDNet average increased 10.92%, 21.13%, 82.97%, 8.55%, and 16.26% than the compared methods, respectively. The source code is available at https://github.com/JasonWong30/DeDNet.

DFG-HCEN: A distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image fusion

In this paper, we propose an unsupervised learning approach for the task of infrared and visible image fusion. This approach is called a distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image Fusion (DFG-HCEN). Instead of using complex fusion rules, DFG-HCEN uses multi-level fusion to achieve fusion results, effectively avoiding information loss during feature extraction. To improve the fusion effect, we designed a distinctive-feature guided module that strengthens the relationship between modules. Moreover, the proposed hierarchical channel enhanced and distinctive-Feature guided module aims to facilitate the fusion framework in efficiently integrating the multilevel complementary features of the source pictures. In addition, we incorporate a hybrid loss method for unsupervised training of the provided DFG-HCEN. The fidelity loss is used to constrain the pixel similarity between the fused result and source images. The application of luminance regularization loss has been shown to be an efficient method for addressing the problem of luminance degradation in fused images. We conducted extensive experiments, including visual examination and quantitative analysis, comparing DFG-HCEN with thirteen other state-of-the-art fusion techniques. The results demonstrate the superiority of DFG-HCEN. Moreover, the extended object detection experiments validate the ability of DFG-HCEN to fully support downstream tasks.

Infrared and Visible Image Fusion (IVF) Using Latent Low-Rank Representation and Deep Feature Extraction Network

Infrared and Visible Image Fusion (IVF) Using Latent Low-Rank Representation and Deep Feature Extraction Network

Photovoltaic fault detection based on infrared and visible image augmentation and fusion

Photovoltaic fault detection based on infrared and visible image augmentation and fusion

Semantic-Aware Fusion Network Based on Super-Resolution.

The aim of infrared and visible image fusion is to generate a fused image that not only contains salient targets and rich texture details, but also facilitates high-level vision tasks. However, due to the hardware limitations of digital cameras and other devices, there are more low-resolution images in the existing datasets, and low-resolution images are often accompanied by the problem of losing details and structural information. At the same time, existing fusion algorithms focus too much on the visual quality of the fused images, while ignoring the requirements of high-level vision tasks. To address the above challenges, in this paper, we skillfully unite the super-resolution network, fusion network and segmentation network, and propose a super-resolution-based semantic-aware fusion network. First, we design a super-resolution network based on a multi-branch hybrid attention module (MHAM), which aims to enhance the quality and details of the source image, enabling the fusion network to integrate the features of the source image more accurately. Then, a comprehensive information extraction module (STDC) is designed in the fusion network to enhance the network's ability to extract finer-grained complementary information from the source image. Finally, the fusion network and segmentation network are jointly trained to utilize semantic loss to guide the semantic information back to the fusion network, which effectively improves the performance of the fused images on high-level vision tasks. Extensive experiments show that our method is more effective than other state-of-the-art image fusion methods. In particular, our fused images not only have excellent visual perception effects, but also help to improve the performance of high-level vision tasks.

GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

SimpliFusion: a simplified infrared and visible image fusion network

SimpliFusion: a simplified infrared and visible image fusion network

A deep learning based relative clarity classification method for infrared and visible image fusion

Infrared and visible image fused(IVIF) results normally suffer from detail loss, noise occurrence, low contrast, and blurred edges. In this paper, a new method is proposed to address the detail loss, low contrast, and blurring issue in IVIF. Specifically, visible images are enhanced by guided filter and high dynamic range compression. Infrared images are normalized by a linear transformation. Then we use blur and clear discrimination to detect salient pixels between infrared and visible images. A fully weight shared multi-path residual neural network is proposed for blur discrimination between infrared and visible image pixels in the same position. Clear pixels are treated as salient pixels, which contribute more to fused images than blur pixels. The output of our proposed network is a binary classification map for blur and clear discrimination, which is treated as our fusion weight map in the fusion stage. To deal with the discontinuous problem, we compute the two distance-transformed maps of the binary classification map and its complementary map. The two distance-transformed maps are used as weight maps to fuse the enhanced infrared and visible images. Finally, we use single scale retinex (SSR) to further enhance our fused images. The experimental results in public IVIF datasets demonstrate the superior performance of our proposed approach over other state-of-the-art methods in terms of both subjective visual quality and objective metrics. The source code is available in https://github.com/eyob12/Multi_path_residual_neural_network_based_IVIF

DSFusion: Infrared and visible image fusion method combining detail and scene information

The goal of image fusion is to combine the complementary features of two images to generate an information-rich fused image. However, taking into account both detail and scene information is difficult for existing image fusion algorithms. Therefore, we propose an infrared and visible image fusion method combining details and scene information (DSFusion). Specifically, we first design a local attention module (LAM), which performs feature extraction on the source image from multiple perspectives in order to better preserve minute detail information. Moreover, in order to distinguish and highlight the differences between the two modal images, we improved the channel attention module. Finally, we design a new loss function that can effectively balance the detail and scene information of the fusion image. Extensive testing on publicly available datasets demonstrates that DSFusion surpasses state of the art in both qualitative and quantitative evaluation. Furthermore, promising outcomes have been obtained in generalization experiments by directly expanding the trained model to other datasets, indicating the model’s excellent generalization capability. The code is available at https://github.com/LKZ1584905069/DSFusion.

Lightweight Infrared and Visible Image Fusion Based on Nested Connections and Res2Net

Lightweight Infrared and Visible Image Fusion Based on Nested Connections and Res2Net

High-level vision task-driven infrared and visible image fusion approach: Progressive Semantic Enhancement based Multi-scale Cross-modality Interactive Network

Infrared-visible image fusion aims to merge data for a more accurate scene representation. Current research focuses mainly on enhancing visual appeal rather than improving performance in high-level vision tasks. To address this gap, we propose the Semantic Enhanced Multi-scale Cross-modality Interactive Image Fusion Network (SeMIFusion). Initially, Multi-scale Cross-modality Feature Fusion (MCFF) module is devised to extract shallow and deep features across different modalities. During feature extraction, Texture Enhancer (TE) and Semantify Enhancer (SE) blocks capture diverse hierarchical features across multi-scale layers, seamlessly integrating into the Semantic Feature Integration (SFI) module for profound semantic information extraction. Furthermore, an Image Scene Reconstruction (ISR) module maintains original image details in fused features, ensuring image fidelity. Additionally, incorporating a visual preservation guiding mask prioritizes retaining visual quality during reconstruction, preventing degradation. Extensive experiments demonstrate our method’s superiority in preserving visual effects and texture details, especially in high-level vision tasks. The source code is available at https://github.com/yyzzttkkjj/SeMIFusion.

ADF‐Net: Attention‐guided deep feature decomposition network for infrared and visible image fusion

AbstractTo effectively enhance the ability to acquire information by making full use of the complementary features of infrared and visible images, the widely used image fusion algorithm is faced with challenges such as information loss and image blurring. In response to this issue, the authors propose a dual‐branch deep hierarchical fusion network (ADF‐Net) guided by an attention mechanism. Initially, the attention convolution module extracts the shallow features of the image. Subsequently, a dual‐branch deep decomposition feature extractor is introduced, where in the transformer encoder block (TEB) employs remote attention to process low‐frequency global features, while the CNN encoder block (CEB) extracts high‐frequency local information. Ultimately, the global fusion layer based on TEB and the local fusion layer based on CEB produce the fused image through the encoder. Multiple experiments demonstrate that ADF‐Net excels in various aspects by utilizing two‐stage training and an appropriate loss function for training and testing.

DDFNet-A: Attention-Based Dual-Branch Feature Decomposition Fusion Network for Infrared and Visible Image Fusion

The fusion of infrared and visible images aims to leverage the strengths of both modalities, thereby generating fused images with enhanced visible perception and discrimination capabilities. However, current image fusion methods frequently treat common features between modalities (modality-commonality) and unique features from each modality (modality-distinctiveness) equally during processing, neglecting their distinct characteristics. Therefore, we propose a DDFNet-A for infrared and visible image fusion. DDFNet-A addresses this limitation by decomposing infrared and visible input images into low-frequency features depicting modality-commonality and high-frequency features representing modality-distinctiveness. The extracted low and high features were then fused using distinct methods. In particular, we propose a hybrid attention block (HAB) to improve high-frequency feature extraction ability and a base feature fusion (BFF) module to enhance low-frequency feature fusion ability. Experiments were conducted on public infrared and visible image fusion datasets MSRS, TNO, and VIFB to validate the performance of the proposed network. DDFNet-A achieved competitive results on three datasets, with EN, MI, VIFF, QAB/F, FMI, and Qs metrics reaching the best performance on the TNO dataset, achieving 7.1217, 2.1620, 0.7739, 0.5426, 0.8129, and 0.9079, respectively. These values are 2.06%, 11.95%, 21.04%, 21.52%, 1.04%, and 0.09% higher than those of the second-best methods, respectively. The experimental results confirm that our DDFNet-A achieves better fusion performance than state-of-the-art (SOTA) methods.

Modality specific infrared and visible image fusion based on multi-scale rich feature representation under low-light environment

Infrared (IR) and Visible (VIS) image fusion is a critical task in computer vision, which aims to produce high-quality fused images by combining two modal images with complementary information. However, current image fusion techniques neglect the severe degradation of texture and details in low-light environments and do not account for human visual perception. In this paper, a multi-scale image fusion algorithm adapted to the low-light environment is proposed. Firstly, the light distribution of the low-light image is estimated and adjusted based on the gray value probability distribution, resulting in an enhanced VIS image with rich details. Simultaneously, considering the imaging characteristics of the two modal images, a superpixel method is utilized to obtain a saliency map for the IR image, and a multi-scale feature extraction scheme is employed to establish a VIS detail map. Finally, we employ entropy to measure the amount of information in the saliency and detail maps to guide the fusion process. Experimental results from qualitative and quantitative evaluations demonstrate that the fused image conforms to human visual perception, maintains a uniform distribution of pixel intensity, and preserves rich texture features at the edge details.

MACCNet: Multiscale Attention and Cross- Convolutional Network for Infrared and Visible Image Fusion

MACCNet: Multiscale Attention and Cross- Convolutional Network for Infrared and Visible Image Fusion

SDFuse: Semantic-injected dual-flow learning for infrared and visible image fusion

Infrared and visible image fusion (IVIF) strives to render fused results that preserve the strengths of the source images (e.g., texture details and thermal highlights) while boosting downstream tasks. However, existing semantic-driven methods typically combine segmentation and fusion tasks via cascade architecture, which fails to sufficiently exploit semantic knowledge. To address this limitation, an innovative semantic-injected network with dual flows is proposed for IVIF, called SDFuse. Concretely, we first apply two pseudo-Siamese feature extraction modules (FEM) to obtain shallow structure features, which involve skip connections to prevent the disappearance of information. Subsequently, dual flows are constructed to support the joint learning paradigm of two tasks. The segmentation flow excavates abundant semantic knowledge from shallow features through fulfilling high-level segmentation task. The fusion flow is introduced to achieve the aggregation of highlights and textures. Note that SGM in fusion flow effectuates the injection of deep semantic knowledge into shallow structural features. Eventually, the region consistency module (RCM) with group normalization leads the network to focus on foreground objects and generate context-consistent fusion images. Our fusion results not only effectively fuse the information of the source image but also have advantages in downstream tasks. Extensive experiments over multiple representative benchmarks prove that SDFuse outperforms current state-of-the-art approaches.

Light-weight Fusion Network for UAV Visible-light and Infrared Images based on Real-time Brain-like Intelligent Computing

Abstract. Multiple sensors equipped on the Unmanned Aerial Vehicle (UAV) enables the acquisition of multi-modal and multi-source remote sensing data. UAV remote sensing usually faces with real-time or near-real-time tasks in complex and highly dynamic environments, such as disaster monitoring, traffic management, border patrol and so on. Under these conditions, the image fusion algorithm needs to be high efficiency, precision and reliability. In this paper, we proposed an intelligent real-time fusion network for UAV multi-source remote sensing data based on AI brain-like chips, and deployed the algorithm on the UAV platform to achieve online high-efficiency computing. Firstly, we have developed a novel image fusion algorithm named SFNet for infrared and visible image fusion based on ShuffleNetv2. Then, we use ZCA and l1-norm to process the remodeled deep feature. The weight maps are generated by bi-cubic interpolation and soft-max operation. Finally, the fused image is reconstructed by weighted-average operation. The proposed SFNet is deployed on the Lynxi KA200 brain computing chip, and a comprehensive inference test is carried out with UAV remote sensing data. Several State-Of-The-Art (SOTA) data fusion algorithms are deployed on the same chip for experimental comparison. The proposed SFNet is proved to have faster inference speed and better feature extraction results on brain-like chips. It is more suitable for real-time UAV remote sensing image fusion tasks.

VCAFusion: An infrared and visible image fusion network with visual perception and cross-scale attention

Infrared and visible image fusion methods aim to combine salient target instances and abundant texture details into fused images. However, due to the interference of harsh conditions, such as dense smoke, fog, and intense light, it is feasible for external interference information to be integrated into the fused image, which seriously affects the image quality. To this end, we propose an infrared and visible image fusion network with visual perception and cross-scale attention modules, termed VCAFusion, which integrates critical information from source images in harsh conditions more efficiently. Specifically, considering that the human eye can identify key information under adverse conditions, we design a visual perception module (VPM), guiding information integration from the perspective of human visual perception. In addition, we propose a cross-scale attention module (CSAM) based on shifted window cross-attention, which aims to capture the long-distance correlation between adjacent scale features, providing more accurate image information for image restoration. Experiments with a variety of datasets reveal that the VCAFusion can adaptively retain image information and further improve image generation ability in harsh conditions.