Thermal Video Sequences Research Articles

The Development of Various Methods for Object Tracking and Classification in Thermal Videos

Nowadays, thermal cameras play a major role because of its temperature-based photography in many applications such as video surveillance, monitoring electronics/electrical machines, forest monitoring, monitoring babies/adult patients, and suspicious object detection. Tracking pedestrians in thermal video is a major task for such applications. Thermal cameras usually create images based on temperature emitted by the object only and not on the lighting conditions and outdoor environment conditions. But still thermal images have constraints like no texture or colour information, more number of dead pixels, low resolution, and noticeable visual colour patterns in case of any temperature variations. So the challenge in tracking pedestrians in thermal videos is tracking objects/pedestrians throughout the video without an identity switch by overcoming these constraints which may mislead the tracking process. To overcome these constraints, the proposed system uses tri feature matrix (TFM) as an object descriptor which is used to uniquely identify and represent objects in thermal images. TFM is represented in more compact way as a triple matrix. It is a simple and accurate descriptor suitable for tracking objects in thermal video sequences without an identity switch. The proposed Pedestrian tracking system uses most of the advantages of thermal cameras by overcoming challenges in thermal videos effectively based on a novel descriptor TFM. The proposed system is evaluated with various data sets, and the results are analysed using true positive, true negative, false negative, false positive, accuracy, precision, recall, F-score, global identity mismatch (GMME) and track matching error (TME). The performance metrics such as accuracy, precision, recall, F-score, GMME and TME are computed as 99%, 100%, 99%, 99%, 2.3%, and 2.1%, respectively. From the observation, it is found that the performance of proposed TFM-based system is significantly improved. The experimental result shows that the proposed system achieved more accurate tracking compared to the conventional methods

Thermal Image Processing for Respiratory Estimation from Cubical Data with Expandable Depth.

As healthcare costs continue to rise, finding affordable and non-invasive ways to monitor vital signs is increasingly important. One of the key metrics for assessing overall health and identifying potential issues early on is respiratory rate (RR). Most of the existing methods require multiple steps that consist of image and signal processing. This might be difficult to deploy on edge devices that often do not have specialized digital signal processors (DSP). Therefore, the goal of this study is to develop a single neural network realizing the entire process of RR estimation in a single forward pass. The proposed solution builds on recent advances in video recognition, capturing both spatial and temporal information in a multi-path network. Both paths process the data at different sampling rates to capture rapid and slow changes that are associated with differences in the temperature of the nostril area during the breathing episodes. The preliminary results show that the introduced end-to-end solution achieves better performance compared to state-of-the-art methods, without requiring additional pre/post-processing steps and signal-processing techniques. In addition, the presented results demonstrate its robustness on low-resolution thermal video sequences that are often used at the embedded edge due to the size and power constraints of such systems. Taking that into account, the proposed approach has the potential for efficient and convenient respiratory rate estimation across various markets in solutions deployed locally, close to end users.

Tensor integrated total variation regularization for thermography NDT of composites

The carbon fiber reinforced polymer (CFRP) is being used frequently in the manufacturing of aerospace, rail, and other mechanical structures, where the optical pulse thermography (OPT)-based non-destructive testing (NDT) is generally used for the quality inspection, However, the output thermal sequences in OPT-based inspection suffer from uneven illumination and high-frequency thermal noise. Consequently, the inspection of defects (debonds) becomes difficult. To remedy it, post-image processing algorithms are generally carried out. The usefulness of such algorithms, however, is limited by the shape-complexity of the CFRP specimen. In this paper, we propose a tensor nuclear norm (TNN)-based low-rank and sparse total variation regularization (TVR) for CFRP debond defect detection. The integrated low-rank and sparse components are jointly and iteratively optimized. The proposed algorithm removes noise and segments/extracts the defects information from the thermal video sequences with improved resolution and contrast. Compared to the general image processing algorithm used for OPT-based NDT testing, the proposed algorithm is faster and in terms of F-score is more accurate.

A cooperative crowdsensing system based on flying and ground vehicles to control respiratory viral disease outbreaks

The massive increase in population density in cities has led to several urban problems, such as an increment of air pollution, traffic congestion, and a faster spread of infectious diseases. With the rapid innovation in the intelligent sensors technology, and its integration into smart vehicles and Unmanned Aerial Vehicles (UAVs), a novel sensing paradigm has been promoted, namely vehicular crowdsensing, which leverages on-board sensors to capture information from the surrounding environment. Collected data are then analyzed to take proper countermeasures. In this paper, we present a smart coordination mechanism between UAVs and ground vehicles (GVs), which sense information like body temperature and breathing rate of people, in order to support a variety of monitoring applications, including discovering the presence of infectious diseases. In our framework, namely GUAVA, aerial and ground vehicles are equipped with GPS devices and thermal cameras to monitor specific geographic areas, detect humans’ vital parameters and, at the same time, discover duplicate data by identifying matching faces in thermal video sequences with the GaussianFace algorithm. The sensing tasks in hard-to-reach places are assigned to UAVs, with the ability to power up wirelessly from the nearest GV and offload the collected monitoring images to it. Simulation results have assessed our proposed framework, showing good performance in terms of distinct Quality of Service (QoS) metrics.

A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences

Infrared-Thermal Imaging is a non-contact mechanism for psychophysiological research and application in Human–Computer Interaction (HCI). Real-time detection of the face and tracking the Regions of Interest (ROI) in the thermal video during HCI is challenging due to head motion artifacts. This paper proposes a three-stage HCI framework for computing the multivariate time-series thermal video sequences to recognize human emotion and provides distraction suggestions. The first stage comprises of face, eye, and nose detection using a Faster R-CNN (region-based convolutional neural network) architecture and used Multiple Instance Learning (MIL) algorithm for tracking the face ROIs across the thermal video. The mean intensity of ROIs is calculated which forms a multivariate time series (MTS) data. In the second stage, the smoothed MTS data are passed to the Dynamic Time Warping (DTW) algorithm to classify emotional states elicited by video stimulus. During HCI, our proposed framework provides relevant suggestions from a psychological and physical distraction perspective in the third stage. Our proposed approach signifies better accuracy in comparison with other classification methods and thermal data-sets.

Pedestrian tracking in thermal videos using TFM (tri-feature matrix)

Nowadays, thermal cameras play a major role because of its temperature-based photography in many applications such as video surveillance, monitoring electronics/electrical machines, forest monitoring, monitoring babies/adult patients, and suspicious object detection. Tracking pedestrians in thermal video is a major task for such applications. Thermal cameras usually create images based on temperature emitted by the object only and not on the lighting conditions and outdoor environment conditions. But still thermal images have constraints like no texture or colour information, more number of dead pixels, low resolution, and noticeable visual colour patterns in case of any temperature variations. So the challenge in tracking pedestrians in thermal videos is tracking objects/pedestrians throughout the video without an identity switch by overcoming these constraints which may mislead the tracking process. To overcome these constraints, the proposed system uses tri feature matrix (TFM) as an object descriptor which is used to uniquely identify and represent objects in thermal images. TFM is represented in more compact way as a triple matrix. It is a simple and accurate descriptor suitable for tracking objects in thermal video sequences without an identity switch. The proposed Pedestrian tracking system uses most of the advantages of thermal cameras by overcoming challenges in thermal videos effectively based on a novel descriptor TFM. The proposed system is evaluated with various data sets, and the results are analysed using true positive, true negative, false negative, false positive, accuracy, precision, recall, F-score, global identity mismatch (GMME) and track matching error (TME). The performance metrics such as accuracy, precision, recall, F-score, GMME and TME are computed as 99%, 100%, 99%, 99%, 2.3%, and 2.1%, respectively. From the observation, it is found that the performance of proposed TFM-based system is significantly improved. The experimental result shows that the proposed system achieved more accurate tracking compared to the conventional methods.

Fast eyes detection in thermal images

In recent years many methods have been proposed for eye detection. In some cases however, such as driver drowsiness detection, lighting conditions are so challenging that only the thermal imaging is a robust alternative to the visible light sensors. However, thermal images suffer from poor contrast and high noise, which arise due to the physical properties of the long waves processing. In this paper we propose an efficient method for eyes detection based on thermal image processing which can be successfully used in challenging environments. Image pre-processing with novel virtual high dynamic range procedure is proposed, which greatly enhances thermal image contrast and allows for more reliable computation of sparse image descriptors. The bag-of-visual-words approach with clustering was selected for final detections. We compare our method with the YOLOv3 deep learning model. Our method attains high accuracy and fast response in real conditions without computational complexity and requirement of a big dataset associated with the deep neural networks. For quantitative analysis a series of thermal video sequences were recorded in which eye locations were manually annotated. Created dataset was made publicly available on our website.

Top-down thermal tracking based on rotatable elliptical motion model for intelligent livestock breeding

Thermal sensors bring a lot of benefits for object tracking due to the robustness to lighting changes over time, which still give a great difficulty to traditional tracking methods. Despite this powerful advantage, the ambiguity due to background clutters and occlusions makes the problem of thermal sensor-based tracking intractable. In this paper, we propose a novel framework for multiple object tracking in livestock breeding spaces using a single thermal sensor. The key idea of the proposed method is to define a new rotatable elliptical motion model, which is suitable for describing complicated motions of animals. Moreover, the proposed local constraint guides our tracker to robustly chase targets even with severe occlusions. Experimental results on various thermal video sequences show the significant improvement for tracking animals compared to other approaches proposed in the literature.

Multispectral particle filter tracking using adaptive decision-based fusion of visible and thermal sequences

One of the main challenges of detection and tracking of objects in video monitoring is the lighting conditions of the scene under surveillance and its variations. Due to the availability of various visible cameras and beyond visible spectrum sensors at low costs, a solution for this challenge is the fusion of different sensors data, especially visible and thermal sensors. However, the main problem is, proposing an efficient fusion of various sensors and determining the reliability of their data to make the most use of the sensor’s information. For this purpose, this paper presents a multispectral particle filter tracking based on an adaptive fusion of visible and thermal data. In the proposed fusion method, a confidence measure is assigned to each sensor’s data. To evaluate the confidence of the sensor’s data in each frame, in the proposed method, two criteria consisting of local mean luminance in the visible spectrum and local contrast in the thermal spectrum are calculated. These criteria are then employed to determine the confidence of each sensor using a fuzzy inference system (FIS). Also, a second confidence measure for the visible spectrum is defined based on the variation of local mean luminance in successive frames. The total confidence of visible and thermal spectrum is then employed for the adaptive fusion of data for particle filter tracking. The proposed method was evaluated quantitatively and qualitatively through several simulations performed on the thermal and visible benchmark video sequences, which include variable and low light conditions. The results were promising to improve the tracking in challenging scenes and demonstrated the effectiveness of the proposed fusion algorithm.

A Vehicle Classification Method in Thermal Video Sequences using both Shape and Local Features

A Vehicle Classification Method in Thermal Video Sequences using both Shape and Local Features

Grayscale-Thermal Tracking via Inverse Sparse Representation based Collaborative Encoding.

Grayscale-thermal tracking has attracted a great deal of attention due to its capability of fusing two different yet complementary target observations. Existing methods often consider extracting the discriminative target information and exploring the target correlation among different images as two separate issues, ignoring their interdependence. This may cause tracking drifts in challenging video pairs. This paper presents a collaborative encoding model called joint correlation and discriminant analysis based inver-sparse representation (JCDA-InvSR) to jointly encode the target candidates in the grayscale and thermal video sequences. In particular, we develop a multi-objective programming to integrate the feature selection and the multi-view correlation analysis into a unified optimization problem in JCDA-InvSR, which can simultaneously highlight the special characters of the grayscale and thermal targets through alternately optimizing two aspects: the target discrimination within a given image and the target correlation across different images. For robust grayscale-thermal tracking, we also incorporate the prior knowledge of target candidate codes into the SVM based target classifier to overcome the overfitting caused by limited training labels. Extensive experiments on GTOT and RGBT234 datasets illustrate the promising performance of our tracking framework.

Regularized kernel matrix decomposition for thermal video multi-object detection and tracking

This paper derives a novel algorithm for joint detection and tracking of multiple moving objects in thermal videos. The problem of determining multiple objects in a frame sequence is formulated as the task of factorizing a properly defined kernel covariance matrix into sparse factors. The support of these factors will point to the indices of the pixels that form each object. A coordinate descent approach is utilized to determine the sparse factors, and extract the object pixels. A centroid pixel is estimated for each object which is subsequently tracked via Kalman filtering. A novel interplay between the sparse kernel covariance factorization scheme along with Kalman filtering is proposed to enable joint object detection and tracking, while a divide and conquer strategy is put forth to reduce computational complexity and enable efficient tracking. Extensive numerical tests on both synthetic data and thermal video sequences demonstrate the effectiveness of the novel approach and superior tracking performance compared to existing alternatives.

Thermal Sensor-Based Multiple Object Tracking for Intelligent Livestock Breeding

Visual object tracking is an essential technique for constructing intelligent livestock management systems. Behavior patterns estimated from the trajectories of animals provide substantial useful information related to estrus cycle, disease prognosis and so on. However, similar colors and shapes between animals often lead to the failure of tracking multiple objects, and the background clutter of the breeding space further makes the problem intractable. In this paper, we propose a novel method for tracking animals using a single thermal sensor. The key idea of the proposed method is to represent the foreground (i.e., animals) easily obtained by a simple thresholding in a thermal frame as a topographic surface, which is very helpful for finding the boundary of each object even in cases with overlapping. Based on the segmentation results derived from morphological operations on the topographic surface, the center positions of all the animals are consistently updated with an efficient refinement scheme that is robust to the abrupt motions of animals. Experimental results using various thermal video sequences demonstrate the efficiency and robustness of our method for tracking animals in a breeding space compared to previous approaches proposed in the literature.

A New Shape Feature for Vehicle Classification in Thermal Video Sequences

Since thermal images represent only the temperature difference between objects and background and they have more blurred edges than color images, the segmented images from them have noisy object boundaries. Therefore, the well-known features developed for color images may not work well with thermal images. To overcome these limitations of thermal images, we propose a novel feature extraction method based on the target trait context (TTC). A robust keypoint detector is also proposed by analyzing the included angle moments of the object boundary points. At each keypoint, we define the boundary shape context and the normalized intensity context, which contain shape information and thermal distribution, respectively, of the object. These two contexts are combined to form a new feature set, the TTC. To validate our proposed feature extraction method, the keypoint repeatability test and the classification performance test were performed and compared with those from the previous methods. The experiment results show that the proposed feature works well for thermal video sequences and outperforms the previous methods in classification performance.

A combined approach of Kullback–Leibler divergence and background subtraction for moving object detection in thermal video

In this work, a robust method for moving object detection in thermal video frames has been proposed by including Kullback–Leibler divergence (KLD) based threshold and background subtraction (BGS) technique. A trimmed-mean based background model has been developed that is capable enough to reduce noise or dynamic component of the background. This work assumed that each pixel has normally distributed. The KLD has computed between background pixel and a current pixel with the help of Gaussian mixture model. The proposed threshold is useful enough to classify the state of each pixel. The post-processing step uses morphological tool for edge linking, and then the flood-fill algorithm has applied for hole-filling, and finally the silhouette of targeted object has generated. The proposed methods run faster and have validated over various real-time based problematic thermal video sequences. In the experimental results, the average value of F1-score, area under the curve, the percentage of correct classification, Matthew’s correlation coefficient show higher values whereas total error and percentage of the wrong classification show minimum values. Moreover, the proposed-1 method achieved higher accuracy and execution speed with minimum false alarm rate that has been compared with proposed-2 as well as considered peer methods in the real-time thermal video.

Research of Vehicle Detection Algorithm in Infrared Thermal Imaging Videos

This paper proposes a new algorithm for counting vehicle information in the infrared thermal imaging video sequences. First, the characteristics of the adjacent frames in video sequences has been analyzed, together with the modeling for the background information and updating background model according to the inter-frame variation. Second, the information of the moving target in the current frame is preliminarily extracted adopting the background subtraction algorithm, it achieves image segmentation and the division of the best detection area according to the imaging features of vehicles in different lanes by using morphological filtering and setting up the detection window. Then, model classification is done using the connected domain tags and template matching for vehicles in detection region. Finally, vehicle information each frame detects is counted by lanes. The experimental results present that the algorithm for dividing the best detection region segmentation image can reduce the miss rate and the false detection rate to the maximum extent. Meanwhile, the algorithm also has good detection accuracy for statistics of the traffic flow and classification of vehicles in video sequences. ©, 2015, Journal of Information and Computational Science. All right reserved.

Effects of image restoration on automatic acquisition of moving objects in thermal video sequences degraded by the atmosphere

We aim to determine the effect of image restoration (deblurring) on the ability to acquire moving objects detected automatically from long-distance thermal video signals. This is done by first restoring the videos using a blind-deconvolution method developed recently, and then examining its effect on the geometrical features of automatically detected moving objects. Results show that for modern (low-noise and high-resolution) thermal imaging devices, the geometrical features obtained from the restored videos better resemble the true properties of the objects. These results correspond to a previous study, which demonstrated that image restoration can significantly improve the ability of human observers to acquire moving objects from long-range thermal videos.

Effects of image restoration on acquisition of moving objects from thermal video sequences degraded by the atmosphere

Remotely sensed videos, captured by high-resolution imagers, are likely to be degraded by the atmosphere. In still images, the degradation sources, which include turbulence and aerosols, mainly cause blur. In video sequences, however, spatiotemporally varying distortions caused by turbulence also become important. These atmospheric degradations reduce image quality and therefore the ability of target acquisition by the observers. The effects of image quality and image restoration (deblurring) on target acquisition in still images were examined previously in several studies. Nevertheless, results obtained in static situations may not be appropriate for dynamic situations (with moving targets), which are frequently more realistic. This work examines the effect of image restoration on the ability of observers to acquire moving objects (such as humans and vehicles) in video sequences. This is done through perception experiments that compare acquisition probabilities in both restored and nonrestored video sequences captured by a remote-sensing thermal imaging system. Results show that image restoration can significantly improve the acquisition probability. These results correspond to the static case. However, unlike the static case, considerably smaller differences were obtained here between the probabilities of target detection and target recognition.