Visible Image Fusion Research Articles

IFICI: Infrared and visible image fusion based on interactive compensation illumination

The goal of infrared and visible image fusion is to generate an informative image that preserves the complementary information of the two types of source images, such as texture details and infrared targets. Existing methods have designed a variety of means to achieve the fusion of image texture and target complementary information. But they ignore the fact that the two types of source images have the complementarity of illumination, especially the infrared image can mainly supplement the information of the low-light visible image. An image fusion method using infrared and visible light to supplement light interactively is proposed. It performs intrinsic image decomposition on the input infrared and visible light images respectively and then fuses the illumination components and material components of the two types of images respectively. Further, to better meet the needs of image fusion, an improved variational model for the decomposition of image intrinsics is proposed. Experimental results demonstrate the effectiveness of the proposed model with its algorithm. Compared with the state-of-the-art methods, the significant advantage of the proposed method is that it has better fusion accuracy and visual perception at the same time, and the comprehensive metric evaluation is the best (9.2% performance improvement). The code is available at https://github.com/skyworkds/IFICI.

A full-scale hierarchical encoder-decoder network with cascading edge-prior for infrared and visible image fusion

Most existing deep learning-based infrared and visible image fusion methods always fail to consider the full-scale long-range correlation and the prior knowledge, resulting in the fused images with low-contrast salient objects and blurred edge details. To overcome these drawbacks, a full-scale hierarchical encoder-decoder network with cascading edge-prior for infrared and visible image fusion is proposed. First, a top-down encoder extracts the hierarchical representations from source image. Then, to inject edge priors into the network and capture the progressive semantic correlations, a triple fusion mechanism is proposed including edge image fusion based on maximum fusion strategy, single-scale shallow layer fusion and full-scale semantic layer fusion based on dual-attention fusion (DAF) strategy. The fused full-scale semantic features (F2SF) are obtained by capturing the long-range affinities of the full-scale. At the same time, a cascading edge-prior branch (CEPB) is designed to embed the fused edge knowledge into fused single-scale shallow features, jointly guiding the decoder to focus on abundant details layer-by-layer on the basis of F2SF, thus recovering the edge and texture details of the fused image well. Finally, a novel loss function consisting of SSIM, intensity and edge loss is constructed to further maintain the network with better edge representation and reconstruction capability. Compared with existing state-of-the-art fusion methods, the proposed method has better performance in terms of both visual evaluation and objective evaluation on public datasets. The source code is available at https://github.com/lxq-jnu/FSFusion.

Crack detection of masonry structure based on thermal and visible image fusion and semantic segmentation

The integration of visible and thermal images has demonstrated the potential ability to enhance crack segmentation accuracy. However, due to the intricate texture of masonry structures and the challenges posed in precisely aligning these cross-modality images, it is necessary to explore pixel-level alignment and develop a comprehensive dataset to enable deep-learning based methods. Therefore, a dataset, Crack900, including five image types, is developed together with a proposed two-step registration to achieve highly accurate pixel-level alignment. In addition, both Train from Scratch and Transfer Learning (TL) strategies are applied on eleven models to investigate the impact of different fused image types. Our findings reveal that the concatenation strategy markedly improves segmentation accuracy, and the performance of TL depends on the compatibility of channel numbers and domain difference between pre-trained and target models. These findings pave the way for further development of cross-modality in masonry crack segmentation methodologies for structural health monitorin.

OMOFuse: An Optimized Dual-Attention Mechanism Model for Infrared and Visible Image Fusion

Infrared and visible image fusion aims to fuse the thermal information of infrared images and the texture information of visible images into images that are more in compliance with people’s visual perception characteristics. However, in the existing related work, the fused images have incomplete contextual information and poor fusion results. This paper presents a new image fusion algorithm—OMOFuse. At first, both the channel and spatial attention mechanisms are optimized by a DCA (dual-channel attention) mechanism and an ESA (enhanced spatial attention) mechanism. Then, an ODAM (optimized dual-attention mechanism) module is constructed to further improve the integration effect. Moreover, a MO module is used to improve the network’s feature extraction capability for contextual information. Finally, there is the loss function ℒ from the three parts of SSL (structural similarity loss), PL (perceptual loss), and GL (gap loss). Extensive experiments on three major datasets are performed to demonstrate that OMOFuse outperforms the existing image fusion methods in terms of quantitative determination, qualitative detection, and superior generalization capabilities. Further evidence of the effectiveness of our algorithm in this study are provided.

ResCCFusion: Infrared and visible image fusion network based on ResCC module and spatial criss-cross attention models

We proposed an infrared and visible image fusion method based on the ResCC module and spatial criss-cross attention models. The proposed method adopts an auto-encoder structure consisting of an encoder network, fusion layers, and a decoder network. The encoder network has a convolution layer and three ResCC blocks with dense connections. Each ResCC block can extract multi-scale features from source images without down-sampling operations and retain as many feature details as possible for image fusion. The fusion layer adopts spatial criss-cross attention models, which can capture contextual information in both horizontal and vertical directions. Attention in these two directions can also reduce the calculation of the attention maps. The decoder network consists of four convolution layers designed to reconstruct images from the feature map. Experiments performed on the public datasets demonstrate that the proposed method obtains better fusion performance on objective and subjective evaluations compared to other advanced fusion methods. The code is available at https://github.com/xiongzhangzzz/ResCCFusion.

Multi-scale convolutional neural networks and saliency weight maps for infrared and visible image fusion

Image fusion is the fusion of multiple images from the same scene to produce a more informative image, and infrared and visible image fusion is an important branch of image fusion. To tackle the issues of diminished luminosity in the infrared target, inconspicuous target features, and blurred texture of the fused image after the fusion of infrared and visible images. This paper introduces a novel effective fusion framework that merges multi-scale Convolutional Neural Networks (CNN) with saliency weight maps. First, the method measures the source image features to estimate the initial saliency weight map. Then, the initial weight map is segmented and optimized using a guided filter before being further processed by CNN. Next, a trained Siamese convolutional network is used to solve the two key problems of activity measure and weight assignment. Meanwhile, a multi-layer fusion strategy is designed to effectively retain the luminance of the infrared target and the texture information in the visible background. Finally, adaptive adjustment of the fusion coefficients is achieved by employing saliency. The experimental results show that the method outperforms the state-of-the-art algorithms in terms of both subjective visual quality and objective evaluation effects.

IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES

Advances in imaging and deep learning have fueled interest in ear biometrics, as the structure of the ear offers unique identification features. Thermal and visible ear images capture different aspects of these features. Thermal images are light-independent, and visible images excel at capturing texture details. Combining these images creates more feature-rich composite images. This study examines the fusion of thermal and visible ear images taken under varying lighting conditions to enhance automatic ear recognition. The image fusion process involved three distinct multiresolution analysis methods: discrete wavelet transform, ridgelet transform, and curvelet transform. Subsequently, a specially designed deep learning model was used for ear recognition. The results of this study reveal that employing the complex-valued curvelet transform and thermal images achieved an impressive recognition rate of 96.82%, surpassing all other methods. Conversely, visible images exhibited the lowest recognition rate of 75.00%, especially in low-light conditions. In conclusion, the fusion of multiple data sources significantly enhances ear recognition effectiveness, and the proposed model consistently achieves remarkable recognition rates even when working with a limited number of fused ear images.

A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict Registration

A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict Registration

DANT-GAN: A dual attention-based of nested training network for infrared and visible image fusion

Existing infrared and visible image fusion methods based on attention mechanisms can perceive the most discriminative regions of the two images. However, the disadvantage is that too many weights are assigned to the area of interest, so some details in the local region are ignored. To solve this problem, we propose a Dual Attention-based Nested Training Network (DANT-GAN), which uses two attention mechanisms to extract and fuse features for local regions and the whole image, respectively. The training process of the entire image is embedded in the training process of the local area of the image, and the training result of the local attention mechanism in each epoch is used as the input to extract and fuse the features of the mixed domain of the whole image. In this way, the information of the attention region can be preserved, and the information lost in the feature extraction phase can be compensated. In addition, the training process is accelerated by increasing the link of generating the teaching network, prompting the model to learn to simulate the fusion ground truth. Experiments show that DANT-GAN can well capture the local and overall attention characteristics of images and describe them on an image. The fusion result can well maintain information in source images, avoids the defect of losing details caused by fusion relying on single attention, and requires fewer computing resources. We compare other state-of-the-art fusion methods on two public datasets, achieving optimal values on SCD, CC, and SSIM. Finally, the effectiveness of the proposed method is verified by ablation experiments.

Lightweight infrared and visible image fusion network with edge-guided dual attention

Lightweight infrared and visible image fusion network with edge-guided dual attention

CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach

Multimodal visual information fusion aims to integrate the multi-sensor data into a single image which contains more complementary information and less redundant features. However the complementary information is hard to extract, especially for infrared and visible images which contain big similarity gap between these two modalities. The common cross attention modules only consider the correlation, on the contrary, image fusion tasks need focus on complementarity (uncorrelation). Hence, in this paper, a novel cross attention mechanism (CAM) is proposed to enhance the complementary information. Furthermore, a two-stage training strategy based fusion scheme is presented to generate the fused images. For the first stage, two auto-encoder networks with same architecture are trained for each modality. Then, with the fixed encoders, the CAM and a decoder are trained in the second stage. With the trained CAM, features extracted from two modalities are integrated into one fused feature in which the complementary information is enhanced and the redundant features are reduced. Finally, the fused image can be generated by the trained decoder. The experimental results illustrate that our proposed fusion method obtains the SOTA fusion performance compared with the existing fusion networks. The codes of our fusion method will be available soon.

W-shaped network: a lightweight network for real-time infrared and visible image fusion

W-shaped network: a lightweight network for real-time infrared and visible image fusion

BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience

In recent years, deep learning research has received significant attention in the field of infrared and visible image fusion. However, the issue of designing loss functions in deep learning-based image fusion methods has not been well-addressed. To tackle this problem, we propose a novel mechanism of utilizing traditional fusion methods as loss functions to guide the training of deep learning models. We incorporate the superior aspects of two traditional methods, namely Guided Filter (GF) and Latent Low-Rank Representation (LatLRR), into the design of the loss function, proposing a fusion method for infrared and visible images that balances both texture and saliency, termed BTSFusion. The proposed network is not only lightweight but also preserves the maximum amount of valuable information in source images. It is worth noting that the complexity of BTSFusion primarily lies in the design of the loss function, which allows it to remain an end-to-end network, as demonstrated by efficiency comparison experiments that highlight the excellent computational efficiency of our algorithm. Furthermore, through subjective observations and objective comparisons, we validated the performance of the proposed method by comparing it with twelve state-of-the-art methods on two public datasets. The source code will be publicly available at https://github.com/YQ-097/BTSFusion.

DCBFusion: an infrared and visible image fusion method through detail enhancement, contrast reserve and brightness balance

DCBFusion: an infrared and visible image fusion method through detail enhancement, contrast reserve and brightness balance

Bounded PCA based Multi Sensor Image Fusion Employing Curvelet Transform Coefficients

The fusion of thermal and visible images acts as an important device for target detection. The quality of the spectral content of the fused image improves with wavelet-based image fusion. However, compared to PCA-based fusion, most wavelet-based methods provide results with a lower spatial resolution. The outcome gets better when the two approaches are combined, but they may still be refined. Compared to wavelets, the curvelet transforms more accurately depict the edges in the image. Enhancing the edges is a smart way to improve spatial resolution and the edges are crucial for interpreting the images. The fusion technique that utilizes curvelets enables the provision of additional data in both spectral and spatial areas concurrently. In this paper, we employ an amalgamation of Curvelet Transform and a Bounded PCA (CTBPCA) method to fuse thermal and visible images. To evidence the enhanced efficiency of our proposed technique, multiple evaluation metrics and comparisons with existing image merging methods are employed. Our approach outperforms others in both qualitative and quantitative analysis, except for runtime performance. Future Enhancement-The study will be based on using the fused image for target recognition. Future work should also focus on this method’s continued improvement and optimization for real-time video processing.

Multi-scale infrared and visible image fusion framework based on dual partial differential equations

Image fusion aims to integrate multiple images that contain complementary information into a single image using an appropriate strategy. The fusion of infrared (IR) and visible (VIS) images can effectively integrate thermal radiation and texture information, which fuses foreground and background information into a fused image. An effective image fusion framework should transfer most of the information from the IR and VIS images to the fused image while minimizing artifacts. This paper introduces a novel multi-scale decomposition framework named Dual PDEs, which decomposes the source images into multiple layers using two different types of partial differential equations (PDEs) and fuses the feature representations at each scale layer separately with an adaptive fusion scheme. The target image is obtained from the multi-scale fused layers. Qualitative and quantitative evaluations show that the proposed framework effectively highlights the prominent areas with fewer artifacts, more explicit boundaries, and better visual effects, which outperforms existing fusion methods.

Infrared and Visible Image Fusion Based on Mask and Cross-Dynamic Fusion

Both single infrared and visible images have respective limitations. Fusion technology has been developed to conquer these restrictions. It is designed to generate a fused image with infrared information and texture details. Most traditional fusion methods use hand-designed fusion strategies, but some are too rough and have limited fusion performance. Recently, some researchers have proposed fusion methods based on deep learning, but some early fusion networks cannot adaptively fuse images due to unreasonable design. Therefore, we propose a mask and cross-dynamic fusion-based network called MCDFN. This network adaptively preserves the salient features of infrared images and the texture details of visible images through an end-to-end fusion process. Specifically, we designed a two-stage fusion network. In the first stage, we train the autoencoder network so that the encoder and decoder learn feature extraction and reconstruction capabilities. In the second stage, the autoencoder is fixed, and we employ a fusion strategy combining mask and cross-dynamic fusion to train the entire fusion network. This strategy is conducive to the adaptive fusion of image information between infrared images and visible images in multiple dimensions. On the public TNO dataset and the RoadScene dataset, we selected nine different fusion methods to compare with our proposed method. Experimental results show that our proposed fusion method achieves good results on both datasets.

Infrared and Visible Image Fusion Using Threshold Segmentation and Weight Optimization

Infrared and Visible Image Fusion Using Threshold Segmentation and Weight Optimization

CLIM: Co-occurrence with Laplacian Intensity Modulation and Enhanced Color Space Transform for Infrared-Visible Image Fusion

Thermal infrared and multispectral visible remote sensing image fusion combines thermal image information with corresponding visible scene content to generate a better representative fused image. Thermal images can distinguish targets using difference in thermal radiation measurements, whereas visible images contain better texture detail in multispectral wavelength bands. The article presents a novel methodology named CLIM to sharpen coarser spatial resolution multispectral remote sensing images using relatively higher spatial resolution broadband thermal infrared image. The boundary-preserving information is extracted from high resolution thermal infrared image using co-occurrence image filter, and is combined with Laplacian of Gaussian based sharpened image to extract salient features for injection. In addition, visible image is transformed to IHS color space, and intensity component is enhanced using CLAHE and inverse transformation to generate enhanced visible image for fusion. The procedure developed is evaluated with Indian Nano Satellite (INS) broadband thermal infrared images available at a spatial resolution of 175 m with same day acquisition MODIS multispectral visible images available at a relatively coarser spatial resolution of 500 m. The nearest acquisition of Landsat-8 thermal infrared images with MODIS multispectral visible images is also used for infrared–visible multi-modal image fusion. The CLIM fused image confirms that distinct features such as dam, ship docking zones and refinery regions, are better demarked and semantically more meaningful in comparison with individual thermal infrared and multispectral visible image. The proposed CLIM approach is compared with, and found to perform better than state-of-the-art image fusion techniques, both visually and quantitatively.

Stretchable hybrid platform‐enabled interactive perception of strain sensing and visualization

AbstractHuman–machine interactive platforms that can sense mechanical stimuli visually and digitally are highly desirable. However, most existing interactive devices cannot satisfy the demands of tactile feedback and extended integration. Inspired by the mechanoluminescence (ML) function of cephalopod skin and the sensitive perception of microcracked slit‐organs, a bioinspired stretchable interactive platform is developed by designing a stretchable poly(styrene‐block‐butadiene‐block‐styrene)/fluorescent molecule (SFM) composite followed by the in situ polymerization of pyrrole (Py) and deposition of carbon nanotubes (CNTs), which possesses a simple multilayered structure and quantitatively senses the applied strains via the variations of digital electrical resistance and visual fluorescence intensity. Using the strain‐dependent microstructures derived from the synergistic interactions of the rigid PPy/CNTs functional layer and SFM, the SFM/PPy/CNTs‐based platforms exhibit excellent strain‐sensing performance manifested by a high gauge factor (GF = 2.64 × 104), wide sensing range (~270%), fast response/recovery time (~155/195 ms), excellent stability (~15,000 cycles at 40% strain), and sensitive ML characteristics under ultraviolet illumination. Benefiting from the novel fusion of digital data and visual images, important applications, including the detection of wrist pulses and human motions, and information dual‐encryption, are demonstrated. This study demonstrates the superiority of advanced structures and materials for realizing superior applications in wearable electronics.