Thermal Infrared Video Research Articles

Fall Detection Method for Infrared Videos Based on Spatial-Temporal Graph Convolutional Network.

The timely detection of falls and alerting medical aid is critical for health monitoring in elderly individuals living alone. This paper mainly focuses on issues such as poor adaptability, privacy infringement, and low recognition accuracy associated with traditional visual sensor-based fall detection. We propose an infrared video-based fall detection method utilizing spatial-temporal graph convolutional networks (ST-GCNs) to address these challenges. Our method used fine-tuned AlphaPose to extract 2D human skeleton sequences from infrared videos. Subsequently, the skeleton data was represented in Cartesian and polar coordinates and processed through a two-stream ST-GCN to recognize fall behaviors promptly. To enhance the network's recognition capability for fall actions, we improved the adjacency matrix of graph convolutional units and introduced multi-scale temporal graph convolution units. To facilitate practical deployment, we optimized time window and network depth of the ST-GCN, striking a balance between model accuracy and speed. The experimental results on a proprietary infrared human action recognition dataset demonstrated that our proposed algorithm accurately identifies fall behaviors with the highest accuracy of 96%. Moreover, our algorithm performed robustly, identifying falls in both near-infrared and thermal-infrared videos.

IBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels

Remote photo-plethysmography (rPPG) has emerged as a non-intrusive and promising physiological sensing capability in human–computer interface (HCI) research, gradually extending its applications in health-monitoring and clinical care contexts. With advanced machine learning models, recent datasets collected in real-world conditions have gradually enhanced the performance of rPPG methods in recovering heart-rate and heart-rate-variability metrics. However, the signal quality of reference ground-truth PPG data in existing datasets is by and large neglected, while poor-quality references negatively influence models. Here, this work introduces a new imaging blood volume pulse (iBVP) dataset of synchronized RGB and thermal infrared videos with ground-truth PPG signals from ear with their high-resolution-signal-quality labels, for the first time. Participants perform rhythmic breathing, head-movement, and stress-inducing tasks, which help reflect real-world variations in psycho-physiological states. This work conducts dense (per sample) signal-quality assessment to discard noisy segments of ground-truth and corresponding video frames. We further present a novel end-to-end machine learning framework, iBVPNet, that features an efficient and effective spatio-temporal feature aggregation for the reliable estimation of BVP signals. Finally, this work examines the feasibility of extracting BVP signals from thermal video frames, which is under-explored. The iBVP dataset and source codes are publicly available for research use.

Multimodal feature fusion for illumination-invariant recognition of abnormal human behaviors

This paper presents a fusion of feature maps extracted from visible and thermal infrared surveillance videos to recognize abnormal human behaviors in low illumination conditions. Abnormal human behaviors often occur at night or in darkness, where a visible camera alone may not capture sufficient visual information for analysis. The proposed scheme uses additional data captured by a thermal infrared camera to enhance information on human appearance and motion. Thermal and visible data are assumed to be informative for analyzing human behaviors, yet they contribute differently to certain areas of video frames. This paper proposes a multimodal feature fusion (MFF) to produce a weight matrix that determines how much thermal or visible input should contribute to obtain optimal performance. Experiment results demonstrate that MFF effectively recognizes abnormal human behaviors in darkness. MFF achieves an overall accuracy of 91.01%, which is at least 4.53% higher than individual thermal or visible data and 2.17% higher than the state-of-the-art fusion-based abnormal human behavior recognition techniques.

Real‐Time Monitoring and Postprocessing of Thermal Infrared Video Images for Sampling and Mapping Groundwater Discharge

AbstractGroundwater discharge along channels can affect stream discharge, chemistry, and ecological communities. Although the spatial distribution of groundwater springs along wide rivers can be investigated by areal thermal infrared (TIR) remote sensing, this technique is difficult to apply to mapping at a high spatial resolution and under riparian tree canopies. We present a real‐time monitoring and postprocessing method of ground‐based TIR video for determining groundwater discharge sampling points and mapping the surface water temperature. We applied this method to mapping two headwater streams in Hokkaido, Japan, in the summer. The first site was a 1.3‐km‐long reach underlain by Pleistocene andesite lava. Almost all of the springs were colder and had a different chemistry compared to that of the stream water, which supports the usefulness of TIR monitoring for determining groundwater discharge zones. Video postprocessing showed that cold groundwater springs were spaced every ∼100 m, and their distribution did not follow the topography. At the second site, cold and warm springs were underlain by Holocene volcanic ash. The cold springs mainly seeped from hyporheic and riparian zones downstream, while warm springs were at the footslope. Some cold springs had much higher solute concentrations than the stream and warm springs, which suggests that the water temperature is useful for inferring sources of groundwater discharge. At this site, the video postprocessing could map not only the locations of the cold springs but also the spatial heterogeneity of the stream temperature associated with groundwater inputs.

KTFEv2: Multimodal Facial Emotion Database and its Analysis

In recent years, the focus of human emotion analysis has gradually shifted towards not only using visible information, but also thermal infrared (IR) information. This requires a great deal of facial emotion data both in visible and thermal IR information. However, most existing databases contain either visible information or posed thermal IR information only. For these reasons, we propose and establish a multimodal facial emotion database including both natural spontaneous visible and thermal IR videos. Beside updating more thermal infrared information, the built dataset in this study also enhances the information of intensity emotions. In which, each emotion is classified into three levels (low, medium, and high). Seven spontaneous emotions from thirty subjects are recorded in the database. Audio and visual stimuli were used to elicit emotions during the experiment. After the standard procedure of collecting data finished, the database has been through the careful annotation and verification procedure. Furthermore, the built database is analyzed by using modern machine learning models such as CNN, ResNet50, YOLO, and using a combination of different models to analyze the dataset. The obtained results are feasible and show that this dataset is useful for use in practice. The results of thermal data analysis provide us with a promising idea for future research on estimating human emotion.

Object detection from UAV thermal infrared images and videos using YOLO models

• 15 models were trained from ground-based images and videos using YOLO networks for UAV-based TIR images and videos detection. • The best and fastest model of the YOLO series models were identified. • UAV TIR videos detection can achieve real-time performance with 40 FPS. Object detection is one of the most crucial tasks in computer vision and remote sensing to identify specific categories of various objects in images. The unmanned aerial vehicle (UAV)-based thermal infrared (TIR) remote sensing multi-scenario images and videos are two important data sources in public security. However, their object detection process is still challenging because of the complicated scene information, coarse resolution compared with the visible videos and lack of public labelled datasets and training models. This study proposed a UAV TIR object detection framework for images and videos. The You Only Look Once (YOLO) models based on Convolutional Neural Network (CNN) architecture were designed to extract features from ground-based TIR images and videos, which were captured by Forward-looking Infrared (FLIR) cameras. The most effective algorithm was finally identified by evaluation metrics and then applied to detect objects on TIR videos from UAVs. Results showed that the highest mean average precision (mAP) of the person and car instances was 88.69% in the validating task. The fastest detection speed achieved 50 frames per second (FPS), and the smallest model size was observed in YOLOv5-s. In the application, the cross-detection performance on persons and cars in UAV TIR videos under a YOLOv5-s model was discussed in terms of the different UAVs’ observation angles and the effectiveness of the YOLO architecture was revealed. This study provides positive support for the qualitative and quantitative evaluation of objection detection from TIR images and videos using deep-learning models.

An AR Geo-Registration Algorithm for UAV TIR Video Streams Based on Dual-Antenna RTK-GPS

In emergency response and disaster rescue, unmanned aerial vehicles (UAVs) onboard thermal infrared (TIR) sensors are an essential means of acquiring ground information in the nighttime working environment. To enable field personnel to make better decisions based on TIR video streams returned from a UAV, the geographic information enhancement of TIR video streams is required. At present, it is difficult for low-cost UAVs to carry high-precision attitude sensors and thus obtain high-precision camera attitude information to meet the enhanced processing requirements of UAV TIR video streams. To this end, this paper proposes an improved Kalman filter algorithm to improve the geographic registration (geo-registration) accuracy by fusing the positioning and heading data from the dual-antenna real-time kinematic global positioning system (RTK-GPS) with onboard internal measurement unit (IMU) data. This method can yield high-precision position and attitude data in real time based on low-cost UAV hardware, based on which high-precision geo-registration results can be obtained. The computational complexity can be reduced compared with video stream feature tracking algorithms. Furthermore, the problem of unstable matching due to the low resolution and texture level of TIR video streams can be avoided. The experimental results prove that the proposed method can reduce the registration error by 66.15%, and significantly improve the geo-registration accuracy of UAV TIR video streams. Thus, it can strongly support the popularization and practicality of the application of augmented reality (AR) technology to low-cost UAV platforms.

Learning Spatio-Temporal Attention Based Siamese Network for Tracking UAVs in the Wild

The popularity of unmanned aerial vehicles (UAVs) has made anti-UAV technology increasingly urgent. Object tracking, especially in thermal infrared videos, offers a promising solution to counter UAV intrusion. However, troublesome issues such as fast motion and tiny size make tracking infrared drone targets difficult and challenging. This work proposes a simple and effective spatio-temporal attention based Siamese method called SiamSTA, which performs reliable local searching and wide-range re-detection alternatively for robustly tracking drones in the wild. Concretely, SiamSTA builds a two-stage re-detection network to predict the target state using the template of first frame and the prediction results of previous frames. To tackle the challenge of small-scale UAV targets for long-range acquisition, SiamSTA imposes spatial and temporal constraints on generating candidate proposals within local neighborhoods to eliminate interference from background distractors. Complementarily, in case of target lost from local regions due to fast movement, a third stage re-detection module is introduced, which exploits valuable motion cues through a correlation filter based on change detection to re-capture targets from a global view. Finally, a state-aware switching mechanism is adopted to adaptively integrate local searching and global re-detection and take their complementary strengths for robust tracking. Extensive experiments on three anti-UAV datasets nicely demonstrate SiamSTA’s advantage over other competitors. Notably, SiamSTA is the foundation of the 1st-place winning entry in the 2nd Anti-UAV Challenge.

Multimodal Information Fusion Approach for Noncontact Heart Rate Estimation Using Facial Videos and Graph Convolutional Network

Heart rate (HR) is a critical signal for reflecting human physical and mental conditions, and it is beneficial for diagnosing neurological and cardiovascular diseases due to its excellent accessibility. However, traditional HR measurement devices have limited usability and convenience. Recent studies have shown that the optical absorption variation of human skin due to blood volume variation in cardiac cycles can be acquired from facial videos and used to estimate HR in a non-contact manner. However, the advanced non-contact HR estimation approaches are based on a single HR information source, resulting in unsatisfactory estimation results due to noise corruption and insufficient information. To address these problems, this paper proposes a multimodal information fusion framework for non-contact HR estimation. First, feature representation maps are used to effectively extract periodic signals from facial visible-light and thermal infrared videos. Then, a temporal-information-aware HR feature extraction network (THR-Net) for encoding discriminative spatiotemporal information from the representation maps is presented. Finally, based on a graph convolution network (GCN), an information fusion model is proposed for feature integration and HR estimation. Experimental and evaluation results of five different metrics on two datasets show that the proposed approach outperforms the state-of-the-art approaches. This paper demonstrates the advantage of multimodal information fusion for non-contact HR estimation.

LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking.

RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a La rge- s cale H igh-diversity [Formula: see text]nchmark for short-term R GBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and densely annotated. LasHeR is highly diverse capturing from a broad range of object categories, camera viewpoints, scene complexities and environmental factors across seasons, weathers, day and night. We conduct a comprehensive performance evaluation of 12 RGBT tracking algorithms on the LasHeR dataset and present detailed analysis. In addition, we release the unaligned version of LasHeR to attract the research interest for alignment-free RGBT tracking, which is a more practical task in real-world applications. The datasets and evaluation protocols are available at: https://github.com/mmic-lcl/Datasets-and-benchmark-code.

An improved mask-RCNN algorithm for UAV TIR video stream target detection

In emergency rescue operations, unmanned aerial vehicles (UAVs) equipped with thermal infrared (TIR) sensors are essential to obtain ground information during nighttime operations. However, existing target detection algorithms mainly consider the detection accuracy but not the processing speed and storage space requirements. Furthermore, current neural network target detection algorithms primarily focus on conventional RGB images and are not optimized for UAV-based TIR video stream data. This paper proposes an improved Mask-RCNN algorithm for target detection in UAV TIR video streams to address current research deficiencies. First, MobileNetV3, which is used to process RGB images, is applied to process TIR data for outdoor emergency rescue operations, significantly increasing the time efficiency of the algorithm. Second, prior knowledge such as the projection model of the airborne camera and the target temperature characteristics in the UAV TIR video stream is utilized to filter the detecting results instead of pre-detection temperature masks. Compared with the original Mask-RCNN algorithm, the improved algorithm increases the processing speed, reduces the storage space requirements, and provides detection performances equal or slightly superior to that before the improvement.

A Survey of Strategies, and Procedures for Detecting Drone Utilizing Sound, Radar, Thermal Infrared Sensor and Video Applications

With the exponential rise in the use of drones anywhere anytime, malicious use by outlaws is increasing as well. This calls for protective, detective, preventive measures to counter these attacks. This paper aims to review literature on drone detection and classification that utilizes a myriad of modalities ranging from using thermal infrared sensors to radar detections. In addition, there is a summary of a detailed discussion on drone classification along with recent progress and development in drone detection using machine learning, all of which is performed in an attempt to identify means to thwart such attacks. Furthermore, some future research directions, related to this new field of study, are discussed.

Eddies in motion: visualizing boundary-layer turbulence above an open boreal peatland using UAS thermal videos

Abstract. High-resolution thermal infrared (TIR) imaging is opening up new vistas in biosphere–atmosphere heat exchange studies. The rapidly developing unmanned aerial systems (UASs) and specially designed cameras offer opportunities for TIR survey with increasingly high resolution, reduced geometric and radiometric noise, and prolonged flight times. A state-of-the-art science platform is assembled using a Matrice 210 V2 drone equipped with a Zenmuse XT2 thermal camera and deployed over a pristine boreal peatland with the aim of testing its performance in a heterogeneous sedge-fen ecosystem. The study utilizes the capability of the UAS platform to hover for prolonged times (about 20 min) at a height of 500 m a.g.l. while recording high frame rate (30 Hz) TIR videos of an area of ca. 430 × 340 m. A methodology is developed to derive thermal signatures of near-ground coherent turbulent structures impinging on the land surface, surface temperature spectra, and heat fluxes from the retrieved videos. The size, orientation, and movement of the coherent structures are computed from the surface temperature maps, and their dependency on atmospheric conditions is examined. A range of spectral and wavelet-based approaches are used to infer the properties of the dominant turbulent scene structures. A ground-based eddy-covariance system and an in situ meteorological setup are used for reference.

A Survey of Strategies, and Procedures for Detecting Drone Utilizing Sound, Radar, Thermal Infrared Sensor and Video Applications

With the exponential rise in the use of drones anywhere anytime, malicious use by outlaws is increasing as well. This calls for protective, detective, preventive measures to counter these attacks. This paper aims to review literature on drone detection and classification that utilizes a myriad of modalities ranging from using thermal infrared sensors to radar detections. In addition, there is a summary of a detailed discussion on drone classification along with recent progress and development in drone detection using machine learning, all of which is performed in an attempt to identify means to thwart such attacks. Furthermore, some future research directions, related to this new field of study, are discussed.

Thermal Infrared Video Stabilization for Aerial Monitoring of Active Wildfires

Measuring wildland fire behavior is essential for fire science and fire management. Aerial thermal infrared (TIR) imaging provides outstanding opportunities to acquire such information remotely. Variables such as fire rate of spread (ROS), fire radiative power (FRP), and fireline intensity may be measured explicitly both in time and space, providing the necessary data to study the response of fire behavior to weather, vegetation, topography, and firefighting efforts. However, raw TIR imagery acquired by unmanned aerial vehicles (UAVs) requires stabilization and georeferencing before any other processing can be performed. Aerial video usually suffers from instabilities produced by sensor movement. This problem is especially acute near an active wildfire due to fire-generated turbulence. Furthermore, the nature of fire TIR video presents some specific challenges that hinder robust interframe registration. Therefore, this article presents a software-based video stabilization algorithm specifically designed for TIR imagery of forest fires. After a comparative analysis of existing image registration algorithms, the KAZE feature-matching method was selected and accompanied by pre- and postprocessing modules. These included foreground histogram equalization and a multireference framework designed to increase the algorithm's robustness in the presence of missing or faulty frames. The performance of the proposed algorithm was validated in a total of nine video sequences acquired during field fire experiments. The proposed algorithm yielded a registration accuracy between 10 and 1000× higher than other tested methods, returned 10× more meaningful feature matches, and proved robust in the presence of faulty video frames. The ability to automatically cancel camera movement for every frame in a video sequence solves a key limitation in data processing pipelines and opens the door to a number of systematic fire behavior experimental analyses. Moreover, a completely automated process supports the development of decision support tools that can operate in real time during an emergency.

Learning Deep Multi-Level Similarity for Thermal Infrared Object Tracking

Existing deep Thermal InfraRed (TIR) trackers only use semantic features to represent the TIR object, which lack the sufficient discriminative capacity for handling distractors. This becomes worse when the feature extraction network is only trained on RGB images. To address this issue, we propose a multi-level similarity model under a Siamese framework for robust TIR object tracking. Specifically, we compute different pattern similarities using the proposed multi-level similarity network. One of them focuses on the global semantic similarity and the other computes the local structural similarity of the TIR object. These two similarities complement each other and hence enhance the discriminative capacity of the network for handling distractors. In addition, we design a simple while effective relative entropy based ensemble subnetwork to integrate the semantic and structural similarities. This subnetwork can adaptive learn the weights of the semantic and structural similarities at the training stage. To further enhance the discriminative capacity of the tracker, we propose a large-scale TIR video sequence dataset for training the proposed model. To the best of our knowledge, this is the first and the largest TIR object tracking training dataset to date. The proposed TIR dataset not only benefits the training for TIR object tracking but also can be applied to numerous TIR visual tasks. Extensive experimental results on three benchmarks demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.

Real-Time Geophysical Monitoring of Particle Size Distribution During Volcanic Explosions at Stromboli Volcano (Italy)

Of all the key parameters needed to inform forecast models, real-time tracking particle size distribution (PSD) of pyroclasts leaving the vent coupled with plume modeling has probably the highest potential for effective management of volcanic crises. This paper presents an algorithm capable of providing syn-emission horizontal size and velocity of particles in real time, converted in mass discharge rates, and its evolution during an explosion, using thermal infrared videos. We present data on explosions that occurred at the SW crater of Stromboli volcano (Italy) in 2012. PSDs and mass eruption rate data, collected at frequencies of 40 Hz, are then coupled with particle and gas speed data collected with traditional image analysis techniques. The dataset is used to quantify the dynamics of the explosions and magma fragmentation. We find that explosive evacuation of magma from a Strombolian conduit during a single explosion proceeds at a constant rate while the explosive dynamics are marked by a pattern that includes an initial transient and short phase until the system stabilizes at equilibrium. These stationary conditions dominate the emission. All explosions begin with a gas jet (onset phase), with maximum recorded vertical velocities above 150 m/s. These high velocities are for small particles carried by the faster-moving gas or pressure wave, and larger particles typically have slower velocities. The gas jets are followed by a particle-loaded plume. The particles increase in number until the explosion dynamics become almost constant (in the stationary phase). Mass eruption rate is either stable or increases during the onset to become stable in the stationary phase. The shearing at the interface between the magma and the gas jets controls fragmentation dynamics and particles sizes. Quantification of the Reynolds and Weber numbers suggest that the fragmentation regime changes during an explosion to affect particle shape. This method relates to lapilli and bomb-dominated emissions associated with low energy primary fragmentation. In general terms, its applicability is limited by the size of individual pixels recorded by the camera, which depends on the detector, the recording distance and the optical system, particle temperature, which has to be significantly higher than the background.

Detection and Tracking of Moving Targets for Thermal Infrared Video Sequences.

The joint detection and tracking of multiple targets from raw thermal infrared (TIR) image observations plays a significant role in the video surveillance field, and it has extensive applied foreground and practical value. In this paper, a novel multiple-target track-before-detect (TBD) method, which is based on background subtraction within the framework of labeled random finite sets (RFS) is presented. First, a background subtraction method based on a random selection strategy is exploited to obtain the foreground probability map from a TIR sequence. Second, in the foreground probability map, the probability of each pixel belonging to a target is calculated by non-overlapping multi-target likelihood. Finally, a generalized labeled multi-Bernoulli (-GLMB) filter is employed to produce the states of multi-target along with their labels. Unlike other RFS-based filters, the proposed approach describes the target state by a pixel set instead of a single point. To meet the requirement of factual application, some extra procedures, including pixel sampling and update, target merging and splitting, and new birth target initialization, are incorporated into the algorithm. The experimental results show that the proposed method performs better in multi-target detection than six compared methods. Also, the method is effective for the continuous tracking of multi-targets.

An Adaptive Fusion Algorithm for Visible and Infrared Videos Based on Entropy and the Cumulative Distribution of Gray Levels

Visible videos captured under different weather conditions may exhibit different characteristics, and thermal infrared videos are easily affected by ambient temperature variations; this sensitivity to environmental conditions makes the fusion of visible and thermal infrared videos a challenge. This paper proposes an adaptive fusion algorithm for visible and infrared videos, and uses cumulative distribution of gray levels and the entropy to adaptively retain infrared-hot targets and visible textures. The original visible and infrared frames are decomposed into two layers, namely, the base layer and the detail layer. The guided filter is employed to decompose frames due to its high efficiency. Two weight maps, one for the infrared base layer and one for the visible base layer, are adaptively generated based on the cumulative distribution of gray levels and the entropy, respectively. The visible base layer and the infrared base layer are fused based on their weight maps. The final fusion result is obtained by combining the fused base layer with the visible detail layer. Experimental results demonstrate that the proposed algorithm can achieve better fusion results compared with state-of-the-art methods.

Integrating puffing and explosions in a general scheme for Strombolian‐style activity

AbstractStrombolian eruptions are among the most common subaerial styles of explosive volcanism worldwide. Distinctive features of each volcano lead to a correspondingly wide range of variations of magnitude and erupted products, but most papers focus on a single type of event at a single volcano. Here, in order to emphasize the common features underlying this diversity of styles, we scrutinize a database from 35 different erupting vents, including 21 thermal infrared videos from Stromboli (Italy), Etna (Italy), Yasur (Vanuatu), and Batu Tara (Indonesia), from puffing, through rapid explosions to normal explosions, with variable ejection parameters and relative abundance of gas, ash, and bombs. Using field observations and high‐speed thermal infrared videos processed by a new algorithm, we identify the distinguishing characteristics of each type of activity and how they may relate and interact. In particular, we record that ash‐poor normal explosions may be preceded and followed by the onset or the increase of the puffing activity, while ash‐rich explosions are emergent, i.e., with inflation of the free surface followed directly by emission of increasingly large gas pockets. Overall, we see that all Strombolian activities form a continuum arising from a common mechanism and are modulated by the combination of two well‐established controls: (1) the length of the bursting gas pocket with respect to the vent diameter and (2) the presence and thickness of a high‐viscosity layer in the uppermost part of the volcanic conduit.