Fusion Of Thermal Images Research Articles

Winter road surface condition classification using convolutional neural network (CNN): visible light and thermal image fusion

During winter, road conditions play a crucial role in traffic flow efficiency and road safety. Icy, snowy, slushy, and wet road conditions reduce tire friction and affect vehicle stability which could lead to dangerous crashes. To keep traffic operations safe, cities spend a significant budget on winter maintenance operations such as snow plowing and spreading of salt and sand. This paper proposes a methodology for automated winter road surface condition classification using convolutional neural network (CNN) and the combination of thermal and visible light cameras. As part of this research, 4244 pairs of visible light and thermal images are captured from pavement surfaces and classified into snowy, icy, wet, and slushy surface conditions. Two single-stream CNN models (visible light and thermal streams) and one dual-stream CNN model are developed. The average F1-Score of dual-stream model is 0.866, 0.935, 0.985, and 0.888 on snowy, icy, wet, and slushy, respectively. The weighted average F1-Score is 0.94.

An effective fuzzy logic and particle swarm optimization based thermal and visible-light image fusion framework using curve-let transform

Thermal and visible-light image fusion is an active and one of the most significant research area in image processing. The result produced through this fusion process can be very much useful in surveillance and military applications especially for improving visibility and situation awareness. This paper proposes an effective fuzzy logic and particle swarm optimization based thermal and visible-light image fusion framework using curve-let transform. A novel fuzzy logic based contrast enhancement method is introduced in this framework for improving the visual appearance of both thermal and visible-light source images. In this framework curve-let transform is used for extracting more multi-scale features from source images. Upon applying curve-let transform on enhanced source images will produce low frequency (approximation) and high frequency (detailed) coefficients. Particle swarm optimization based fusion strategy has been used for producing more featured fused approximation coefficients. Correspondingly Max fusion strategy has been used for generating more textures based fused detailed coefficients. Finally upon applying inverse curve-let transform generates final fused image. The obtained simulation results clearly demonstrate that our framework has shown superior results compared to other fusion methods both in terms of subjective and objective assessments.

DepthFuseNet: an approach for fusion of thermal and visible images using a convolutional neural network

We address an approach called “DepthFuseNet” for the fusion of thermal and visible images using convolutional neural networks (CNN). The thermal image acquires radiating energy of the sensed objects and hence it can easily distinguish the objects from its background. However, the visible image (i.e., the image acquired within the range of visible wavelength of electromagnetic spectrum) provides a more visual context of the objects with high spatial resolution. Due to this complement nature of thermal and visible images, it is always an interest of the community to combine those two images to obtain more meaningful information from the individual source images. In DepthFuseNet method, features are extracted from given source images using CNN architecture, and they are integrated using the different fusion strategies. The auto-weighted sum fusion strategy performs better than that obtained using the other existing methods. To reduce the complexity of the architecture, we use depthwise convolution in the network. The experimental evaluation demonstrates that the proposed method exhibits salient features from the source images, and hence it performs better than the other state-of-the-art fusion methods in terms of qualitative and quantitative assessments.

Visible-to-Thermal Transfer Learning for Facial Landmark Detection

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.

ABiFN: Attention‐based bi‐modal fusion network for object detection at night time

Camera-based object detection in low-light/night-time conditions is a fundamental problem because of insufficient lighting. So far, a mid-level fusion of RGB and thermal images is done to complement each other's features. In this work, an attention-based bi-modal fusion network is proposed for a better object detection in the thermal domain by integrating a channel-wise attention module. The experimental results show that the proposed framework improves the mAP by 4.13 points on the FLIR dataset.

Video Flame and Smoke Based Fire Detection Algorithms: A Literature Review

This review is focused on video flame and smoke based fire detection algorithms for both indoor and outdoor environments. It analyzes and discusses them in a taxonomical manner for the last two decades. These are mainly based on handcraft features with or without classifiers and deep learning approaches. The separate treatment is provided for detecting flames and smoke. Their static and dynamic characteristics are elaborated for the handcraft feature approach. The blending of the obtained features from these characteristics is the focus of most of the research and these concepts are analyzed critically. A fusion of both visible and thermal images leading to multi-fusion and multimodal approaches have conversed. It is a step towards obtaining accurate detection results and how the handcraft feature approach tackles the problems of flame and smoke detection, as well as their weaknesses are discussed which are still not solved. Some of these weaknesses can be tackled by developing a technology based on artificial intelligence named deep-learning. Its taxonomical literature study with a focus on the flame and smoke detection is presented. The strengths and weaknesses of this approach are discussed with possible solutions. The latest trend in literature which focuses on the hybrid approach utilizing both handcraft feature, and deep learning approaches is discussed. This approach aims to minimize the weaknesses still present in the current systems.

Facial Recognition for Drunk People Using Thermal Imaging

Face recognition using thermal imaging has the main advantage of being less affected by lighting conditions compared to images in the visible spectrum. However, there are factors such as the process of human thermoregulation that cause variations in the surface temperature of the face. These variations cause recognition systems to lose effectiveness. In particular, alcohol intake causes changes in the surface temperature of the face. It is of high relevance to identify not only if a person is drunk but also their identity. In this paper, we present a technique for face recognition based on thermal face images of drunk people. For the experiments, the Pontificia Universidad Católica de Valparaíso-Drunk Thermal Face database (PUCV-DTF) was used. The recognition system was carried out by using local binary patterns (LBPs). The LBP features were obtained from the bioheat model from thermal image representation and a fusion of thermal images and a vascular network extracted from the same image. The feature vector for each image is formed by the concatenation of the LBP histogram of the thermogram with an anisotropic filter and the fused image, respectively. The proposed technique has an average percentage of 99.63% in the Rank-10 cumulative classification; this performance is superior compared to using LBP in thermal images that do not use the bioheat model.

Optimal fusion aided face recognition from visible and thermal face images

This paper proposes a novel Eigen face recognition that is aided by fusion of visible and thermal face images to improve the face recognition accuracy. We adopt three different fusion schemes where in the face information is fused by the optimal weights obtained by different optimization algorithms. The first two fusion approaches operate in the dual tree discrete wavelet transform (DT-DWT), while the third one operates in the Curvelet transform (CT) domain. We employ particle swarm optimization (PSO), self-tuning particle swarm optimization (STPSO) and brain storm optimization algorithm (BSO) to find optimal fusion coefficients. The proposed fusion aided face recognition approaches are evaluated through extensive experiments using OCTVBS benchmark face database and the Eigen face detection methodology. Simulation results show that proposed face recognition techniques have significant performance improvement in recognition accuracy suggesting fusion aided face recognition approach that deserves further study and consideration whenever high recognition accuracy is desired.

Deep Visible and Thermal Image Fusion for Enhanced Pedestrian Visibility.

Reliable vision in challenging illumination conditions is one of the crucial requirements of future autonomous automotive systems. In the last decade, thermal cameras have become more easily accessible to a larger number of researchers. This has resulted in numerous studies which confirmed the benefits of the thermal cameras in limited visibility conditions. In this paper, we propose a learning-based method for visible and thermal image fusion that focuses on generating fused images with high visual similarity to regular truecolor (red-green-blue or RGB) images, while introducing new informative details in pedestrian regions. The goal is to create natural, intuitive images that would be more informative than a regular RGB camera to a human driver in challenging visibility conditions. The main novelty of this paper is the idea to rely on two types of objective functions for optimization: a similarity metric between the RGB input and the fused output to achieve natural image appearance; and an auxiliary pedestrian detection error to help defining relevant features of the human appearance and blending them into the output. We train a convolutional neural network using image samples from variable conditions (day and night) so that the network learns the appearance of humans in the different modalities and creates more robust results applicable in realistic situations. Our experiments show that the visibility of pedestrians is noticeably improved especially in dark regions and at night. Compared to existing methods we can better learn context and define fusion rules that focus on the pedestrian appearance, while that is not guaranteed with methods that focus on low-level image quality metrics.

Human authentication based on fusion of thermal and visible face images

In recent past, considerable amount of research has been done to increase the performance of a face authentication system in uncontrolled environment such as illumination. However, the performance has not been improved significantly since visible face images are dependent of illumination. To overcome the limitation of visible face images, researchers are using infrared (IR) face images. However, it also does not completely independent of illumination. Image fusion of visible and thermal face images is an alternative in research community nowadays. In this work, a fusion method is introduced to fuse visible and IR images for face authentication. The proposed fusion method relies on translation invariant A-trous wavelet transform and fractal dimension using differential box counting method. Five popular fusion metrics namely, ratio of spatial frequency error, normalized mutual information, edge information, universal image quality index, extended frequency comparison index are considered to measure the effectiveness of the proposed fusion algorithm quantitatively over four state-of-the-art methods. A new similarity measure is also proposed to check how close a fused face image from others are. All the experiments are performed on three databases namely, IRIS benchmark face database, UGC-JU face database and SCface face database. All the results depict that the proposed fusion method along with similarity measure for face authentication outperforms all the four state-of-the-art methods in terms of accuracy, precision and recall.

Spectral Total-Variation Local Scale Signatures for Image Manipulation and Fusion.

We propose a unified framework for isolating, comparing and differentiating objects within an image. We rely on the recently proposed total-variation transform, yielding a continuous, multi-scale, fully edge-preserving, local descriptor, referred to as spectral total-variation local scale signatures. We show and analyze several useful merits of this framework. Signatures are sensitive to size, local contrast and composition of structures; are invariant to translation, rotation, flip and linear illumination changes; and texture signatures are robust to the underlying structures. We prove exact conditions in the 1D case. We propose several applications for this framework: saliency map extraction for fusion of thermal and optical images or for medical imaging, clustering of vein-like features and size-based image manipulation.

Thermal Augmented Expression Recognition.

Visible facial images provide geometric and appearance patterns of facial expressions and are sensitive to illumination changes. Thermal facial images record facial temperature distribution and are robust to light conditions. Therefore, expression recognition is enhanced by visible and thermal image fusion. In most cases, only visible images are available due to the widespread popularity of visible cameras and the high cost of thermal cameras. Thus, we propose a novel visible expression recognition method by using thermal infrared (IR) data as privileged information, which is only available during training. Specifically, we first learn a deep model for visible images and thermal images. Then we use the learned feature representations to train support vector machine (SVM) classifiers for expression classification. We jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Thermal IR images during training are then exploited to construct better facial representations and expression classifiers from visible images. We extend the proposed thermal augmented expression recognition method for partially unpaired data, acknowledging that visible images and thermal images maybe not be recorded synchronously. Experimental resulton the MAHNOB laughter database demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal IR images' supplementary role for visible facial expression recognition during training to obtain better facial representations and a better visible expression classifier. The proposed thermal augmented expression recognition method achieves state-of-the-art expression recognition performance for both paired and unpaired facial images.

THERMAL AND VISIBLE SATELLITE IMAGE FUSION USING WAVELET IN REMOTE SENSING AND SATELLITE IMAGE PROCESSING

Abstract. Multimodal remote sensing approach is based on merging different data in different portions of electromagnetic radiation that improves the accuracy in satellite image processing and interpretations. Remote Sensing Visible and thermal infrared bands independently contain valuable spatial and spectral information. Visible bands make enough information spatially and thermal makes more different radiometric and spectral information than visible. However low spatial resolution is the most important limitation in thermal infrared bands. Using satellite image fusion, it is possible to merge them as a single thermal image that contains high spectral and spatial information at the same time. The aim of this study is a performance assessment of thermal and visible image fusion quantitatively and qualitatively with wavelet transform and different filters. In this research, wavelet algorithm (Haar) and different decomposition filters (mean.linear,ma,min and rand) for thermal and panchromatic bands of Landast8 Satellite were applied as shortwave and longwave fusion method . Finally, quality assessment has been done with quantitative and qualitative approaches. Quantitative parameters such as Entropy, Standard Deviation, Cross Correlation, Q Factor and Mutual Information were used. For thermal and visible image fusion accuracy assessment, all parameters (quantitative and qualitative) must be analysed with respect to each other. Among all relevant statistical factors, correlation has the most meaningful result and similarity to the qualitative assessment. Results showed that mean and linear filters make better fused images against the other filters in Haar algorithm. Linear and mean filters have same performance and there is not any difference between their qualitative and quantitative results.

Fusion of Visible and Thermal Images Using a Directed Search Method for Face Recognition

A new image fusion algorithm based on the visible and thermal images for face recognition is presented in this paper. The new fusion algorithm derives the benefit from both the modalities images. The proposed fusion process is the weighted sum of thermal and visible face information with two weighting factors [Formula: see text] and [Formula: see text], respectively. The weighting factors are calculated using a directed search algorithm automatically. The proposed fusion framework is evaluated through extensive experiments using UGC-JU face database. Experiments are of three fold. Firstly, individual modalities images are used separately for human face recognition. Secondly, fused face images using the proposed method are used for recognition purpose. The highest level of accuracy achieved by using the proposed method is about 98.42%. Lastly, the three existing fusion methods are applied on the same face database for comparison with the results of the proposed method. All the results demonstrate significant performance improvements in recognition over individual modalities and some of the existing fusion approaches, suggesting that fusion is a viable approach that deserves further study and consideration.

Visible and Infrared Thermal Image Fusion Algorithm Based on Self-Adaptive Reference Image

The technology of fusing visible images and infrared thermal images is one of the most important researching fields in terms of high-performance night vision technology home and abroad Using this kind of technology can efficiently improve people’s ability of target-detecting and scene’s comprehension. The commonly used color transfer algorithm belongs to single reference image’s global color transfer, which results in huge influence on the color of fusion image. Thus, this kind of method cannot adapt to different kinds of scenes. In this essay, we focus on environmental adaptation problems of regular color transfer in color image fusion algorithm base on YUV color space. After analyzing the mean and standard deviation of typical scenes’ UV channel, we find that it has obvious classified characteristics. As a result, we propose a method of constructing adaptive reference image based on mean and standard deviation of U and V channels, which makes visible and infrared thermal image fusion algorithm better adapt to environment and fusion images’ color is natural. Furthermore, the handling capacity is small and it exerts little impact on the speed of algorithm real-time hardware processing. To conclude, it is a kind of practical natural color fusion processing algorithm.

Swarm intelligence based optimisation in thermal image fusion using dual tree discrete wavelet transform

In this paper, we propose a particle swarm optimised image fusion framework in Discrete Wavelet Transform domain (DWT) that combines the thermal image with the visual image to obtain a single informative fused image. Dual tree Discrete Wavelet Transform (DT-DWT) is applied for feature selection and Particle Swarm Optimisation technique is used to obtain the optimised image. In the fusion process, an optimised weighting factor has been used to construct a new composite image with maximum entropy and minimum Root Mean Square Error. We investigate the fusion performance of the proposed fusion algorithm with distorted input images by adding Gaussian white noise and blurring. The fusion results are compared with existing multi-resolution based fusion techniques such as Laplacian Pyramid, Gradient Pyramid, Ratio of Laplacian Pyramid and Shift Invariant Discrete Wavelet Transform. The quality of the fused image is assessed using seven quality metrics and the results indicate that our proposed method outperforms the state-of-the-art methods in both subjective and objective quality.

Implementation of an algorithm for forming a color image from monochrome images of visible and near infrared cameras in the YCbCr color space

We consider a simplified algorithm for fusion of greyscale visible and thermal images presented in false colors in the de-correlated YCbCr color space. The color gamut is then brought to daylight conditions using a color transfer algorithm that provides the same luminosity of the resulting gray fusion and color fusion images. It is shown that the parallel computing on the graphics card performs real-time video fusion with a frame size of up to 1024×768 pixels and a frame rate of 30 Hz.

The Fusion of the Visual and Thermal Images on the Basis of Determining the Image Fragments which Contain Essential Details

The aim of the following study was to develop a procedure which guarantees the data fusion of thermal and visual images. The first stage of the proposed algorithm consisted of images acquisition which guaranteed that the same parts of images represented the same parts of the observed terrain. The second stage depended on previous information about the searched object features. Two different situations were considered herein. In the case when we had the searched object’s feature vector for both representations of a searched object, we could conduct the pattern recognition for each image. It was conducted separately for visual and thermal images. In this way, we obtained the important parts of the images which should be represented in a fused image. The other case examined in the paper, considered the situation in which we did not have the formalised information about the object. In this case, it was necessary to analyse whole images in order to define the potential parts of the images where the object could be found. This analysis should be helpful for an operator to indicate the parts of the images where there are some artefacts which can be the elements of the searched object. Therefore, in this case, the second stage of the algorithm consisted in calculating the local features of the images. These features constituted grey scale gradient computed for the pixels inside the aperture. This study presented the examples of the fused images obtained by means of the developed method.

A novel template protection scheme for multibiometrics based on fuzzy commitment and chaotic system

In recent years, biometrics template protection has been extensively studied and lots of schemes have been proposed. However, most of them have not considered the forgery, large difference of intra-class and the security of unimodal biometrics leakage. And there is no multibiometrics template scheme based on the fusion of dual iris, thermal and visible face images. In this paper, a novel multibiometrics template protection scheme based on fuzzy commitment and chaotic system, and the security analysis approach for unimodal biometrics leakage are proposed. Firstly, the thermal face images are captured to overcome the forgery. Then, the fuzzy commitment is generated from the corporation of error correcting code (ECC) and the fusion binary features. Additionally, the dual iris feature vectors are encrypted via the chaotic system, and the score level fusion based on Aczel-Alsina triangular-norm (AA T-norm) is implemented to acquire the final verification performance. Finally, the entropy of both mutlibiometrics and unimodal information leakage is analyzed to show the security of the proposed approach. The experimental tests are conducted on a virtual multibiometrics database, which merges the challenging CASIA-Iris-Thousand and the NVIE face database. The verification performance decreases from EER of \(3 \times 10^{-2}\) to \(1.163 \times 10^{-1}\) %, but the multibiometrics template security is enhanced from 80.53 to 167.80 bits based on BCH ECC (1,023, 123, 170).

Adaptive polar transform and fusion for human face image processing and evaluation

AbstractHuman face processing and evaluation is a problem due to variations in orientation, size, illumination, expression, and disguise. The goal of this work is threefold. First, we aim to show that the variant of polar transformation can be used to register face images against changes in pose and size. Second, implementation of fusion of thermal and visual face images in the wavelet domain to handle illumination and disguise and third, principal component analysis is applied in order to tackle changes due to expressions up to a particular extent of degrees. Finally, a multilayer perceptron has been used to classify the face image. Several techniques have been implemented here to depict an idea about improvement of results. Methods started from the simplest design, without registration; only combination of PCA and MLP as a method for dimensionality reduction and classification respectively to the range of adaptive polar registration, fusion in wavelet transform domain and final classification using MLP. A consistent increase in recognition performance has been observed. Experiments were conducted on two separate databases and results yielded are very much satisfactory for adaptive polar registration along with fusion of thermal and visual images in the wavelet domain.