Infrared Image Fusion Research Articles

Temperature Field Prediction Model for Zinc Oxide Rotary Volatile Kiln Based on the Fusion of Thermodynamics and Infrared Images

The timely and accurate measurement of temperature field is of great significance for the low-carbon and high-efficiency operation of the zinc oxide rotary volatile kiln (ZORVK). Due to the large axial length, closed internal space and complex reaction mechanism, it is difficult to measure complete temperature field data. In this study, a novel temperature field prediction model based on the fusion of thermodynamics and infrared images is proposed for the first time. First, a thermodynamic model involving complex chemical reaction heat is established. Then, an industrial infrared thermal imager is developed. In order to obtain high-precision solid fluidized bed temperature from infrared images, a temperature extraction algorithm based on infrared image processing and YOLOv5s is designed. Finally, a parameter optimization model is built by minimizing the error between the predicted temperature of the thermodynamic model and extracted actual temperature. Moreover, five common thermodynamic parameters applicable to any ZORVK are extracted, which reduces the optimization cost of unknown parameters. Results show that our proposed fusion model can effectively improve the prediction accuracy and optimization efficiency of the pure thermodynamic model.

MrFDDGAN: Multireceptive Field Feature Transfer and Dual Discriminator-Driven Generative Adversarial Network for Infrared and Color Visible Image Fusion

Most of the previous infrared and visible image fusion methods based on deep learning were on the basis of grayscale images and used the single convolution kernel receptive field to extract deep features, which would inevitably cause information loss in the process of feature transfer. Accordingly, this paper proposes a new generative adversarial network based on multi-receptive field feature transfer and dual discriminators, which is named MrFDDGAN. It is applied to infrared and color visible image fusion. Firstly, three different receptive fields are used to extract multi-scale and multi-level deep features of multi-modal images on three channels, so as to obtain significant features of source images from multi-view and multi-perspective in a more comprehensive way. Secondly, a feature interaction module is introduced in the encoder to realize the information interaction and information pre-fusion among the three feature channels. Thirdly, the multi-level features fused in the encoder are cascaded with the deep features in the decoder to enhance feature transfer and feature reuse. Finally, this experiment adopts a dual-discriminator adversarial network structure to keep the balance of infrared image intensity and visible image texture preserved by fusion results. And qualitative and quantitative experimental analyses are carried out on two public datasets of grayscale infrared as well as visible images and one dataset of infrared and color visible images. The experimental results prove that the proposed MrFDDGAN algorithm has a better subjective visual effect and objective performance than the existing state-of-the-art fusion methods.

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models.

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce Delta E as a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code is available at https://github.com/GeoVectorMatrix/Dif-Fusion.

FECFusion: Infrared and visible image fusion network based on fast edge convolution.

The purpose of infrared and visible image fusion is to integrate the complementary information from heterogeneous images in order to enhance their detailed scene information. However, existing deep learning fusion methods suffer from an imbalance between fusion performance and computational resource consumption. Additionally, fusion layers or fusion rules fail to effectively combine heteromodal feature information. To address these challenges, this paper presents a novel algorithm called infrared and visible image fusion network base on fast edge convolution (FECFusion). During the training phase, the proposed algorithm enhances the extraction of texture features in the source image through the utilization of structural re-parameterization edge convolution (RECB) with embedded edge operators. Subsequently, the attention fusion module (AFM) is employed to sufficiently fuze both unique and public information from the heteromodal features. In the inference stage, we further optimize the training network using the structural reparameterization technique, resulting in a VGG-like network architecture. This optimization improves the fusion speed while maintaining the fusion performance. To evaluate the performance of the proposed FECFusion algorithm, qualitative and quantitative experiments are conducted. Seven advanced fusion algorithms are compared using MSRS, TNO, and M3FD datasets. The results demonstrate that the fusion algorithm presented in this paper achieves superior performance in multiple evaluation metrics, while consuming fewer computational resources. Consequently, the proposed algorithm yields better visual results and provides richer scene detail information.

YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer

Infrared and visible image fusion is aims to generate a composite image that can simultaneously describe the salient target in the infrared image and texture details in the visible image of the same scene. Since deep learning (DL) exhibits great feature extraction ability in computer vision tasks, it has also been widely employed in handling infrared and visible image fusion issue. However, the existing DL-based methods generally extract complementary information from source images through convolutional operations, which results in limited preservation of global features. To this end, we propose a novel infrared and visible image fusion method, i.e., the Y-shape dynamic Trans- former (YDTR). Specifically, a dynamic Transformer module (DTRM) is designed to acquire not only the local features but also the significant context information. Furthermore, the proposed network is devised in a Y-shape to comprehensively maintain the thermal radiation information from the infrared image and scene details from the visible image. Considering the specific information provided by the source images, we design a loss function that consists of two terms to improve fusion quality: a structural similarity (SSIM) term and a spatial frequency (SF) term. Extensive experiments on mainstream datasets illustrate that the proposed method outperforms both classical and state- of-the-art approaches in both qualitative and quantitative assess- ments. We further extend the YDTR to address other infrared and RGB-visible images and multi-focus images without fine- tuning, and the satisfactory fusion results demonstrate that the proposed method has good generalization capability. Our code is available at <uri>https://github.com/tthinking/YDTR</uri>.

LDRepFM: A Real-Time End-to-End Visible and Infrared Image Fusion Model Based on Layer Decomposition and Re-Parameterization

Visible (VIS) and infrared (IR) image fusion (VIF) is a technique used to synthesize the fused image of high visual perception. Existing fusion methods typically work by discovering the commons underlying the two modalities and fusing them in the common space. However, these methods often ignore the modality differences such as fuzzy details in the IR image and their well-designed architectures also lead to slow fusion speed. To address these issues, we propose a real-time end-to-end VIF model based on layer decomposition and re-parameterization (LDRepFM). This model is composed of a Layer Decomposition Guidance Network (LDGNet) and a Re-parameterization Fusion Network (RepFNet). Firstly, the LDGNet is used to alleviate the visual quality degradation of the fused image by decomposing the IR image into structural layer and fuzzy layer. Secondly, in order to achieve a favorable trade-off between the fusion speed and evaluation metrics, the RepFNet is utilized to decouple the training-time multi-branch and inference-time plain architecture. Thirdly, the structural layer that has been decomposed by LDGNet is utilized in constructing the guidance fusion loss, which is aimed at optimizing RepFNet. Finally, experiments conducted on the publicly available TNO, RoadScene, M3FD, and RegDB datasets demonstrate the performance of the proposed method to be comparable to the state-of-the-art in terms of both visual effect and quantitative metrics. The code is publically available at https://github.com/luming1314/LDRepFM.

Retracted: Wireless Sensor System of UAV Infrared Image and Visible Light Image Registration Fusion

Retracted: Wireless Sensor System of UAV Infrared Image and Visible Light Image Registration Fusion

Infrared and visible image fusion using a guiding network to leverage perceptual similarity

In infrared (IR) and visible image fusion, visual appearance of fused images produced by an end-to-end fusion network relies on a loss function that defines the desired properties of the fusion results. The previous approaches mainly focus on measuring the similarity of an output image with respective IR and visible images using pixel-wise loss functions such as mean square error or structural similarity. This can lead, however, to preserve only local information in the fused images, and it may lose semantic information in the scene. In this work, we propose a new joint loss function for the image fusion, which considers visual quality and subsequent applications simultaneously. Our joint loss function guides the fusion network by transferring the knowledge of a pre-trained network to leverage perceptual similarity. The contrast loss function serves to prevent the reduction of global and local image contrasts in the fused image. The conventional data fidelity is also considered in the proposed function. Then, the joint loss function is used to train our U-Net like fusion network, a hybrid of convolutional and transformer blocks. Experimental results show that the proposed method preserves detailed scene structure from the both of source images and demonstrates the better quantitative results. Furthermore, the proposed fusion schemes can be used for combining multi-exposure and multi-focus image pairs. To show the further effectiveness of IR and visible image fusion, we apply the results of fusion methods for downstream tasks so that the proposed method improves the cases of only using original modalities.

Visual Navigation Algorithm for Night Landing of Fixed-Wing Unmanned Aerial Vehicle

In the recent years, visual navigation has been considered an effective mechanism for achieving an autonomous landing of Unmanned Aerial Vehicles (UAVs). Nevertheless, with the limitations of visual cameras, the effectiveness of visual algorithms is significantly limited by lighting conditions. Therefore, a novel vision-based autonomous landing navigation scheme is proposed for night-time autonomous landing of fixed-wing UAV. Firstly, due to the difficulty of detecting the runway caused by the low-light image, a strategy of visible and infrared image fusion is adopted. The objective functions of the fused and visible image, and the fused and infrared image, are established. Then, the fusion problem is transformed into the optimal situation of the objective function, and the optimal solution is realized by gradient descent schemes to obtain the fused image. Secondly, to improve the performance of detecting the runway from the enhanced image, a runway detection algorithm based on an improved Faster region-based convolutional neural network (Faster R-CNN) is proposed. The runway ground-truth box of the dataset is statistically analyzed, and the size and number of anchors in line with the runway detection background are redesigned based on the analysis results. Finally, a relative attitude and position estimation method for the UAV with respect to the landing runway is proposed. New coordinate reference systems are established, six landing parameters, such as three attitude and three positions, are further calculated by Orthogonal Iteration (OI). Simulation results reveal that the proposed algorithm can achieve 1.85% improvement of AP on runway detection, and the reprojection error of rotation and translation for pose estimation are 0.675∘ and 0.581%, respectively.

MDFN: Mask deep fusion network for visible and infrared image fusion without reference ground-truth

A single infrared image or visible image cannot clearly present texture details and infrared information of the scene in poor illumination, bad weather, or other complex conditions. Thus, it is necessary to fuse the infrared and visible images into one image. In this paper, we propose a novel deep fusion architecture for fusing visible and infrared images without any reference ground-truth. Different from existing deep image fusion methods which directly output the fused images, a weight score corresponding to each pixel is estimated by our network to determine the contributions of two source images. This strategy transfers the valuable information in source images to the fused image. Considering the salient thermal radiation information in the infrared image, a mask of the infrared image is generated and used to preserve valuable contents in the infrared and visible images for the fused image. Furthermore, a hybrid loss is designed to make the fused image consistent with two source images. On account of the weight estimation, the mask strategy, and the hybrid loss, the images fused by our proposed method jointly maintain the thermal radiation and texture details, achieving state-of-the-art performance compared with existing fusion approaches. Our code is publicly available at https://github.com/NlCxg/MDFN.

Maritime Infrared and Visible Image Fusion Based on Refined Features Fusion and Sobel Loss

Infrared (IR) and visible image fusion has become an important technology for maritime environment awareness. In this study, we employed refined features fusion and Sobel loss for maritime IR and visible image fusion. First, we used the encoder in Densefuse net to transform the image to be fused from the pixel domain to the feature domain. Then, the features’ information was accurately measured, and the refined features were fused and sent to the decoder to reconstruct the fused images. Mixed loss, commonly employed in image fusion, eliminates the effect of different losses but also introduces hyperparameters. We innovatively used Sobel loss to measure the input and fused images to reduce the hyperparameters in the loss function. Finally, experiments were performed, and our algorithm achieved good fusion results in the qualitative and quantitative evaluations.

Visible and Infrared Image Fusion-Based Image Quality Enhancement with Applications to Space Debris On-Orbit Surveillance

The increasing amount of space debris in recent years has greatly threatened space operation. In order to ensure the safety level of spacecraft, space debris perception via on-orbit visual sensors has become a promising solution. However, the perception capability of visual sensors largely depends on illumination, which tends to be insufficient in dark environments. Since the images captured by visible and infrared sensors are highly complementary in dark environments, a convolutional sparse representation-based visible and infrared image fusion algorithm is proposed in this paper to expand the applicability of visual sensors. In particular, the local contrast measure is applied to obtain the refined weight map for fusing the base layers, which is more robust in a dark space environment. The algorithm can settle two significant problems in space debris surveillance, namely, improving the signal-noise ratio in a noise space environment and preserving more detailed information in a dark space environment. A space debris dataset containing registered visible and infrared images has been purposely created and used for algorithm evaluation. Experimental results demonstrate that the proposed method in this paper is effective for enhancing image qualities and can achieve favorable effects compared to other state-of-the-art algorithms.

IMGAN: Infrared and visible image fusion using a novel intensity masking generative adversarial network

Fusion methods based on a deep convolutional neural network enable to catch visible details and infrared intensity. However, it is insufficient for most fusion models to extract the hierarchical features of visible images, leading to the loss of detailed information such as texture features in the final fused images. This paper proposes an unsupervised infrared and visible fusion model based on a novel intensity masking adversarial learning network (IMGAN). A designed deep generator with a residual dense attention module in the network can not only make full use of all characteristics of convolution layers but also achieve the more accurate extraction of visible detailed texture. Two discriminators are employed to balance the fused information, making it possible for the fused image to reserve much more information from source images. An infrared intensity masking loss is also proposed to reserve more detailed texture features in the weaker infrared area of the visible image. Furthermore, a large number of compared experiments with the infrared and visible benchmark datasets demonstrate the superiority of our IMGAN than the state-of-the-art methods in terms of both human eye visual perception and quantitative evaluation metrics. Our IMGAN can retain the infrared target information well and secure more detailed textures from the visible image.

Wireless Sensor System of UAV Infrared Image and Visible Light Image Registration Fusion

The application of multisource sensors to drones requires high-quality images to ensure it. In simultaneous interpreting two or more multisensor images based on the same scene or target, the image obtained by the UAV sensor is limited by the imaging time and the shooting angle. The images obtained may not be aligned in the spatial position, thus affecting the fusion effect. Therefore, different sensor images must be registered before image fusion. During the shooting process of the drone imaging sensor, imaging angle, and environmental conditions, the obtained various sensor images will have rotation, translation, and other deformations in the spatial position so that they do not reach the spatial position. Therefore, it is impossible to directly perform image fusion directly. Therefore, before the multisensor image fusion, the image registration process must be completed to ensure that the two images are aligned in space. This paper analyzes the principles; based on the principles of the Powell search algorithm and improved walking algorithm, an algorithm combining Powell and improved walking algorithm is proposed. This paper also studies several traditional image neutrosophic fusions. The algorithm combines the fusion optimization algorithm proposed in this paper greatly reduces the calculation speed and improves the performance of the optimization algorithm and success rate.

Feature Matching technology of Visible and Infrared Fusion Image based on VIFB

In view of the needs of all-time navigation tasks, visual navigation technology based on visible and infrared images has received extensive attention. At present, visible and infrared fusion technology is mostly used in remote sensing detection, target recognition and other fields, but it has not been widely used in the field of navigation and lacks a representative technology evaluation system. Therefore, a feature matching performance evaluation framework for visible and infrared fusion image based on the VIFB framework is proposed in this paper. A variety of typical indoor and outdoor target scenes are designed to test 20 visible and infrared image fusion algorithms. Then the ORB feature matching method is used to test the feature matching effect of image pairs before and after fusion. A variety of qualitative and quantitative evaluation indexes are designed to evaluate the VIFB framework. The feature matching performance of the images before and after processing is comprehensively evaluated. The experiments show that the fusion image can effectively deal with poor lighting and smoke occlusion. In various test scenarios, NSCT_SR, HMSD_GF and LatLRR algorithms show better feature matching performance, and CNN, MST_SR algorithms better integrate visible details and infrared salient structural information. The images fused by ADF, DLF, FPDE algorithms have good grayscale equalization effect.

A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism.

Pixel-level image fusion is an effective way to fully exploit the rich texture information of visible images and the salient target characteristics of infrared images. With the development of deep learning technology in recent years, the image fusion algorithm based on this method has also achieved great success. However, owing to the lack of sufficient and reliable paired data and a nonexistent ideal fusion result as supervision, it is difficult to design a precise network training mode. Moreover, the manual fusion strategy has difficulty ensuring the full use of information, which easily causes redundancy and omittance. To solve the above problems, this paper proposes a multi-stage visible and infrared image fusion network based on an attention mechanism (MSFAM). Our method stabilizes the training process through multi-stage training and enhances features by the learning attention fusion block. To improve the network effect, we further design a Semantic Constraint module and Push–Pull loss function for the fusion task. Compared with several recently used methods, the qualitative comparison intuitively shows more beautiful and natural fusion results by our model with a stronger applicability. For quantitative experiments, MSFAM achieves the best results in three of the six frequently used metrics in fusion tasks, while other methods only obtain good scores on a single metric or a few metrics. Besides, a commonly used high-level semantic task, i.e., object detection, is used to prove its greater benefits for downstream tasks compared with single-light images and fusion results by existing methods. All these experiments prove the superiority and effectiveness of our algorithm.

Visible and Near Infrared Image Fusion Using Base Tone Compression and Detail Transform Fusion

This study aims to develop a spatial dual-sensor module for acquiring visible and near-infrared images in the same space without time shifting and to synthesize the captured images. The proposed method synthesizes visible and near-infrared images using contourlet transform, principal component analysis, and iCAM06, while the blending method uses color information in a visible image and detailed information in an infrared image. The contourlet transform obtains detailed information and can decompose an image into directional images, making it better in obtaining detailed information than decomposition algorithms. The global tone information is enhanced by iCAM06, which is used for high-dynamic range imaging. The result of the blended images shows a clear appearance through both the compressed tone information of the visible image and the details of the infrared image.

DSG-Fusion: Infrared and visible image fusion via generative adversarial networks and guided filter

The goal of infrared and visible image fusion is to generate an informative image which preserves texture details and infrared targets. Most generative adversarial network (GAN) based methods take a concatenation of two source images as the input, hence the extracted feature is insufficient for preserving background and detail information. To this end, we propose a novel GAN based fusion framework, termed as double-stream guided filter network (DSG-Fusion). Given the diverse modalities of infrared and visible images, the generator of DSG network extracts features of two images through two independent data flows. To address the problem of extracting representative background information and force the DSG network focus on details, we integrate guided filter into the generator to obtain base and detail layers of source images. The base layers are concatenated with the corresponding source images to extract deeper features, while detail layers participate in the fusion procedure directly. Thus, DSG-Fusion can extract texture and intensity information sufficiently, and more background and detail information are preserved. Furthermore, a DSG loss consisting of intensity and structural similarity (SSIM) is designed to impel the network to generate a better fused image. Extensive experimental results show that DSG-Fusion achieves better performance comparing with five representative methods.

CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decomposition

In this paper, we propose a novel method for visible and infrared image fusion by decomposing feature information, which is termed as CUFD. It adopts two pairs of encoder–decoder networks to implement feature map extraction and decomposition, respectively. On the one hand, the shallow features of the image contain abundant information while the deep features focus more on extracting the thermal targets. Thus, we use an encoder–decoder network to extract both shallow and deep features. Unlike existing methods, both of the shallow and deep features are used for fusion and reconstruction with different emphases. On the other hand, the infrared and visible features of the same layer have both similarities and differences. Therefore, we train the other encoder–decoder network to decompose the feature maps into common and unique information based on their similarities and differences. After that, we apply different fusion rules according to the flexible requirements. This operation is more beneficial to retain the significant feature information in the fusion results. Qualitative and quantitative experiments on publicly available TNO and RoadScene datasets demonstrate the superiority of our CUFD over the state-of-the-art.

DDGANSE: Dual-Discriminator GAN with a Squeeze-and-Excitation Module for Infrared and Visible Image Fusion

Infrared images can provide clear contrast information to distinguish between the target and the background under any lighting conditions. In contrast, visible images can provide rich texture details and are compatible with the human visual system. The fusion of a visible image and infrared image will thus contain both comprehensive contrast information and texture details. In this study, a novel approach for the fusion of infrared and visible images is proposed based on a dual-discriminator generative adversarial network with a squeeze-and-excitation module (DDGANSE). Our approach establishes confrontation training between one generator and two discriminators. The goal of the generator is to generate images that are similar to the source images, and contain the information from both infrared and visible source images. The purpose of the two discriminators is to increase the similarity between the image generated by the generator and the infrared and visible images. We experimentally demonstrated that using continuous adversarial training, DDGANSE outputs images retain the advantages of both infrared and visible images with significant contrast information and rich texture details. Finally, we compared the performance of our proposed method with previously reported techniques for fusing infrared and visible images using both quantitative and qualitative assessments. Our experiments on the TNO dataset demonstrate that our proposed method shows superior performance compared to other similar reported methods in the literature using various performance metrics.