Infrared-visible Fusion Research Articles

Vision based nighttime pavement cracks pixel level detection by integrating infrared visible fusion and deep learning

Accurate detection of pavement cracks is an important task in road maintenance and safety management. However, accurate segmentation of road cracks at night is still challenging due to light conditions. In this study, an automatic segmentation method for nighttime road cracks based on infrared-visible fusion and deep learning is proposed. First, a fusion method of infrared and visible light images is proposed to improve the visibility of road cracks under low light conditions. Afterwards, a deep learning network integrating a dynamic sparse attention mechanism is proposed to segment the cracks in the enhanced road images. In this study, a dataset of infrared and visible light images of nighttime road cracks is acquired to test the effectiveness and sophistication of the proposed method. The results show that the proposed method can achieve accurate segmentation of nighttime pavement cracks (77.89 % mIoU、85.68 % mPA、97.74 % Accuracy、87.53 % Precision、81.55 % Recall、84.44 % F1-score) and better than the existing segmentation models (Unet, Pspnet, DeepLabv3+). Integration of the proposed method into an unmanned inspection robot helps to achieve 24/7 pixel-level pavement crack detection.

Trans2Fuse: Empowering image fusion through self-supervised learning and multi-modal transformations via transformer networks

Image fusion enhances a single image by integrating information from multiple sources with complementary data. Present end-to-end fusion methods often face overfitting or intricate parameter tuning due to inadequate task-specific training data. To address this, two-stage approaches utilize encoder–decoder networks trained on extensive natural image datasets, yet suffer from limited performance due to domain disparities. In this work, we devise a novel encoder–decoder fusion framework and introduce a self-supervised scheme based on destruction–reconstruction. This approach facilitates task-specific feature learning by proposing three auxiliary tasks: pixel intensity non-linear transformation for multi-modal fusion, brightness transformation for multi-exposure fusion, and noise transformation for multi-focus fusion. By randomly selecting one task during model training, we mutually reinforce different fusion tasks, enhancing network generalizability. We innovate an encoder combining Convolutional Neural Network (CNN) and Transformer to extract both local and global features. Rigorous evaluations against 11 traditional and deep learning-based methods span four benchmark datasets: infrared-visible fusion, medical fusion, multi-exposure fusion, and multi-focus fusion. Comprehensive assessments, encompassing nine metrics from diverse viewpoints, consistently demonstrate the superior performance of our approach in all scenarios. We will make our code, datasets, and fused images publicly available.

Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception

Infrared-visible fusion has great potential in night-vision enhancement for intelligent vehicles. The fusion performance depends on fusion rules that balance target saliency and visual perception. However, most existing methods do not have explicit and effective rules, which leads to the poor contrast and saliency of the target. In this paper, we propose the SGVPGAN, an adversarial framework for high-quality infrared-visible image fusion, which consists of an infrared-visible image fusion network based on Adversarial Semantic Guidance (ASG) and Adversarial Visual Perception (AVP) modules. Specifically, the ASG module transfers the semantics of the target and background to the fusion process for target highlighting. The AVP module analyzes the visual features from the global structure and local details of the visible and fusion images and then guides the fusion network to adaptively generate a weight map of signal completion so that the resulting fusion images possess a natural and visible appearance. We construct a joint distribution function between the fusion images and the corresponding semantics and use the discriminator to improve the fusion performance in terms of natural appearance and target saliency. Experimental results demonstrate that our proposed ASG and AVP modules can effectively guide the image-fusion process by selectively preserving the details in visible images and the salient information of targets in infrared images. The SGVPGAN exhibits significant improvements over other fusion methods.

AR-Aided Smart Sensing for In-Line Condition Monitoring of IGBT Wafer

This paper describes an augmented reality (AR)-aided smart sensing technique for in-line condition monitoring of insulated-gate bipolar transistor (IGBT) wafers. A series of signal processing algorithms are applied for enabling sensor intelligence. Based on electromagnetic infrared–visible fusion (IVF), a supplementary palpable three-dimensional thermography layer is integrated with an IGBT wafer in real world environment. Before the IVF, independent component analysis is implemented to identify defects in the wafer. The proposed AR-aided smart sensing technique enhances user's perception and interaction between the industrial systems and the surrounding world. In contrast to conventional sensor techniques, it provides nondestructive testing and evaluation based high-throughput in-line condition monitoring method. The advantages of noncontact and time efficient of this smart sensing technique potentially bring huge benefit to yield management and production efficiency. AR-aided smart sensing can improve the productivity, quality, and reliability of power electronic materials and devices, as well as in other industrial applications.

A DT-CWT-based infrared-visible image fusion method for smart city

Following the development of smart city, informative images play a more and more important role in recognition, detection, and perception. As an efficient way, image fusion technique integrates information from multiple images. Multi-scale transform (MST) and sparse representation (SR) are widely used in infrared-visible image fusion. Traditional MST-based fusion methods are difficult to represent all features of source images. At the same time, traditional SR-based fusion methods do not consider morphological information of image features in dictionary learning processes. To overcome the defects of both MST- and SR-based fusion methods, this paper presents a infrared-visible image fusion framework by combining double-tree complex wavelet transform (DT-CWT) and SR. The source images are decomposed and clustered into high and low-pass bands by DT-CWT. The high-pass bands are fused by the sum modified-Laplacian (SML). The low-pass bands are fused by SR-based approach. The fused high- and low-pass bands are integrated and reconstructed by DT-CWT to form the final fused image. Comparing with five mainstream image fusion solutions, the proposed fusion framework can achieve state-of the-art performance in infrared-visible fusion images.