Thermal Video Data Research Articles

Respiratory Rate Estimation from Thermal Video Data Using Spatio-Temporal Deep Learning.

Thermal videos provide a privacy-preserving yet information-rich data source for remote health monitoring, especially for respiration rate (RR) estimation. This paper introduces an end-to-end deep learning approach to RR measurement using thermal video data. A detection transformer (DeTr) first finds the subject's facial region of interest in each thermal frame. A respiratory signal is estimated from a dynamically cropped thermal video using 3D convolutional neural networks and bi-directional long short-term memory stages. To account for the expected phase shift between the respiration measured using a respiratory effort belt vs. a facial video, a novel loss function based on negative maximum cross-correlation and absolute frequency peak difference was introduced. Thermal recordings from 22 subjects, with simultaneous gold standard respiratory effort measurements, were studied while sitting or standing, both with and without a face mask. The RR estimation results showed that our proposed method outperformed existing models, achieving an error of only 1.6 breaths per minute across the four conditions. The proposed method sets a new State-of-the-Art for RR estimation accuracy, while still permitting real-time RR estimation.

Fully automated unsupervised learning approach for thermal camera calibration and an accurate COVID-19 human temperature tracking

During the past three years, people have suffered greatly from what the World Health Organization called the emerging COVID-19. The world lacked the means and methods for early detection of this virus. Several traditional methods were used for detecting the virus, such as thermometers, remote thermal detection guns, and other conventional methods. Most of these systems monopolized making profits or selling their camera products, with prices for these cameras equipped with temperature detection systems exceeding three thousand dollars. An unsupervised model for real-time detection of thermal face skin temperature is proposed. Despite the scarcity and limited availability of thermal video data, we found and used a database created at Nazarbayev University in Nur-Sultan, Kazakhstan, which contains clips of thermal video and RGB video. The two different videos were calibrated, and the unity was measured by two metrics: SSIM and correlation. Four methods of registration were used to achieve perfect congruence, and congruence was also measured through the two previous metrics. The K-means method was then used to extract clusters, and functions for post-processing were built. The thermal face skin was extracted by multiplying the binary face mask with the thermal face, and the temperature of the face was calculated by taking the average values of the thermal face skin pixels and converting them from Fahrenheit to Celsius. Satisfactory results were obtained, with some temperatures detected within the normal range, others below the normal range, and others higher than this rate.

Distracted driving recognition based on functional connectivity analysis between physiological signals and perinasal perspiration index

Automatic detection of distracted driving is essential to ensure safety of drivers. In this paper, a novel set of features were extracted from thermal and physiological signals in order to detect and recognize distraction of drivers. Thermal video data which measured the temperature of different areas of the face, heart rate, breathing rate and behavioural signals were used while various types of distractions including cognitive, emotional and sensory-motor were applied to the subjects. The proposed discriminator features were extracted by different functional connectivity methods between the perinasal perspiration extracted from thermal images of the face and physiological variables of heart rate and breathing rate. After feature extraction, binary classification methods were applied to detect the distractions. The results showed that using functional connectivity features significantly increased the accuracy of distraction detection system (up to 99.16% with a highly acceptable F1 score of 0.99). Hence, the proposed model significantly improved (p-value<0.001) the detection of driver’s distraction compared to previous studies. Functional connectivity between perinasal perspiration and breathing signals was the most informative feature for distraction detection purpose. This feature set explained almost all modulations in physiological changes by distractions. For the first time, we designed a system capable of recognizing types of distractions including sensory-motor, cognitive, emotional distractions and also normal state. We used the same feature set to recognize different types of distractions by using three-class and four-class classifiers. The suggested methods distinguished three types of distractions with the best accuracy of 81.94% related to cognitive, sensory-motor distraction and no-distraction states. We also tried to discriminate two types of cognitive distractions, analytic and mathematical distractions. The recognition system classified two types of cognitive distractions with an accuracy of 91.78%. The results suggest that there is important and complementary information in the connectivity between facial temperature signals and physiological variables for distraction detection and recognition.

Machine learning method to predict and analyse transient temperature in submerged arc welding

Heat distribution in the submerged arc welding (SAW) process has a significant impact on the quality of welds. In this paper, a machine learning method is proposed to predict and analyze temperature in the SAW process. Thermal video data is obtained from an infrared camera at the bottom surface of the workpiece. Programs written in MATLAB are used to extract the temperature history and to generate transient isotherm using image processing techniques. The evolution of the transient isotherm throughout the heating and cooling cycle is analyzed quantitatively. The experimental datasets are suitably prepared for use in machine learning. Prediction of local temperature, temperature–time curve, and temperature field map is made through the proposed machine learning method. Various additional characteristics of the temperature field map are analyzed and predicted. The proposed method is experimentally validated and the predicted results agree closely with the experimental data.

Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data.

The contactless estimation of vital signs using conventional color cameras and ambient light can be affected by motion artifacts and changes in ambient light. On both these problems, a multimodal 3D imaging system with an irritation-free controlled illumination was developed in this work. In this system, real-time 3D imaging was combined with multispectral and thermal imaging. Based on 3D image data, an efficient method was developed for the compensation of head motions, and novel approaches based on the use of 3D regions of interest were proposed for the estimation of various vital signs from multispectral and thermal video data. The developed imaging system and algorithms were demonstrated with test subjects, delivering a proof-of-concept.

Principal component analysis for dynamic thermal video analysis

Objective: Current methods for the analysis of dynamic thermal video data from human participants, such as the estimation of mean temperatures from manually selected regions of interest, are rudimentary and provide very limited insight on temperature dynamics. This work proposes a method for the decomposition and analysis of dynamic thermal data to identify different sources of temporal temperature changes in the data. Methods: Principal component analysis (PCA) was applied to thermal video data to identify different sources of changes in temperature. The implemented algorithms were applied on dynamic thermal data of a thermally passive, inanimate object as well as thermal video data of the plantar aspect of human feet. Results: Different sources of temperature variations, consisting of a combination of passive surface cooling, environmental processes and physiological processes were identified. The passive cooling of the skin, typically observed during acclimatization, was noted to last over 60 min, much longer than the five to 20 min durations suggested in the literature. The decomposition that results from the proposed method uncovers underlying temperature dynamics that would typically not emerge from conventional analysis approaches since these would be overshadowed by this passive cooling component. Significance: This method of decomposition of the temporal changes in dynamic thermal data can provide a deeper understanding of the processes driving the temperature changes. The code for the developed algorithms has been made available online in the form of Python and Matlab functions, together with the results presented in this paper, and can be accessed from https://github.com/gaucijean/ThermoSuite.

Principal Component Analysis of Dynamic Thermography Data from Pregnant and Non-Pregnant Women.

In this work we propose a novel approach for the analysis ofdynamic thermography data based on the application of principal component analysis to thermal video data. The proposed approach is applied to thermal video recordings of the abdominal region of pregnant and non-pregnant female participants, and reveals consistent temperature trends across participants that to date have not been reported. Both for the pregnant and non-pregnant participants, the first principal component was found to describe approximately 80% of the total variance, and when combined, the first three principal components explained more than 90% of the total variance. The presence of consistent temporal components across participants is indicative of common passive as well as active underlying mechanisms thatgive rise to the observed temperature patterns. The outcome of this investigation supports further development and application of the proposedmethod in obstetrics and other medical fields.

Unmanned Aerial Vehicles (UAVs) and Artificial Intelligence Revolutionizing Wildlife Monitoring and Conservation

Surveying threatened and invasive species to obtain accurate population estimates is an important but challenging task that requires a considerable investment in time and resources. Estimates using existing ground-based monitoring techniques, such as camera traps and surveys performed on foot, are known to be resource intensive, potentially inaccurate and imprecise, and difficult to validate. Recent developments in unmanned aerial vehicles (UAV), artificial intelligence and miniaturized thermal imaging systems represent a new opportunity for wildlife experts to inexpensively survey relatively large areas. The system presented in this paper includes thermal image acquisition as well as a video processing pipeline to perform object detection, classification and tracking of wildlife in forest or open areas. The system is tested on thermal video data from ground based and test flight footage, and is found to be able to detect all the target wildlife located in the surveyed area. The system is flexible in that the user can readily define the types of objects to classify and the object characteristics that should be considered during classification.

Pedestrian Crosswalk Safety at Nonsignalized Crossings During Nighttime: Use of Thermal Video Data and Surrogate Safety Measures

This paper proposes a methodology to evaluate crosswalk pedestrian safety at nighttime by using surrogate safety measures derived from thermal video data. The methodology is illustrated for two unsignalized crosswalk locations in downtown Montreal, Quebec, Canada. Video recordings from a thermal camera were used to compare nighttime and daytime safety conditions with surrogate safety measures that included vehicle approaching speed, postencroachment time (PET), yielding compliances, and conflict rates. A disaggregate measure of pedestrian exposure that excludes noninteracting road users is also proposed. A thermal camera was used to alleviate issues pertaining to low visibility at night for video analysis when road users, especially pedestrians, are difficult to track. The results showed that the thermal-video–based methodology could effectively collect interaction data at night regardless of lighting conditions. Through the use of thermal video data and the methodology proposed in this paper, the interactions between crossing pedestrians and motor vehicles, with related measures such as PET and speed, could be analyzed to evaluate the effect of different crosswalk treatments on pedestrian safety in low-visibility conditions.

WELD: Weighted Low-rank Decomposition for Robust Grayscale-Thermal Foreground Detection

This paper investigates how to fuse grayscale and thermal video data for detecting foreground objects in challenging scenarios. To this end, we propose an intuitive yet effective method called weighted low-rank decomposition (WELD), which adaptively pursues the cross-modality low-rank representation. Specifically, we form two data matrices by accumulating sequential frames from the grayscale and the thermal videos, respectively. Within these two observing matrices, WELD detects moving foreground pixels as sparse outliers against the low-rank structure background and incorporates the weight variables to make the models of two modalities complementary to each other. The smoothness constraints of object motion are also introduced in WELD to further improve the robustness to noises. For optimization, we propose an iterative algorithm to efficiently solve the low-rank models with three subproblems. Moreover, we utilize an edge-preserving filtering-based method to substantially speed up WELD while preserving its accuracy. To provide a comprehensive evaluation benchmark of grayscale-thermal foreground detection, we create a new data set including 25 aligned grayscale-thermal video pairs with high diversity. Our extensive experiments on both the newly created data set and the public data set OSU3 suggest that WELD achieves superior performance and comparable efficiency against other state-of-the-art approaches.

Detailed analysis of particle launch velocities, size distributions and gas densities during normal explosions at Stromboli

Using high frame rate (33Hz) thermal video data we describe and parameterize the emission and ascent dynamics of a mixed plume of gas and particles emitted during a normal explosion at Stromboli (Aeolian Islands, Italy). Analysis of 34 events showed that 31 of them were characterized by a first phase characterized by an initial diffuse spray of relatively small (lapilli-sized) particles moving at high velocities (up to 213ms−1; average 66–82ms−1). This was followed, typically within 0.1s, by a burst comprising a mixture of ash and lapilli, but dominated by larger bomb-sized particles, moving at lower exit velocities of up to 129ms−1, but typically 46ms−1. We interpret these results as revealing initial emission of a previously unrecorded high velocity gas-jet phase, to which the lapilli are coupled. This is followed by emission of slower moving larger particles that are decoupled from the faster moving gas-phase. Diameters for particles carried by the gas phase are typically around 4cm, but can be up to 9cm, with the diameter of the particles carried by the gas jet (D) decreasing with increased density and velocity of the erupted gas cloud (ρgas and Ugas). Data for 101 particles identified as moving with the gas jet during 32 eruptions allow us to define a new relation, whereby Ugas=Uparticle+a [ρgasD]b. Here, Uparticle is the velocity of bombs whose motion is decoupled from that of the gas cloud, and a and b are two empirically-derived coefficients. This replaces the old relation, whereby Ugas=Uparticle+k D; a relation that requires a constant gas density for each eruption. This is an assumption that we show to be invalid, with gas density potentially varying between 0.04kgm−3 and 9kgm−3 for the 32 cases considered, so that k varies between 54m1/2s−1 and 828m1/2s−1, compared with the traditionally used constant of 150m1/2s−1.