Remote Heart Rate Estimation Research Articles

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning.

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

DeepFake Detection with Remote Heart Rate Estimation Using 3D Central Difference Convolution Attention Network

Objective: As GAN-based deepfakes have become increasingly mature and realistic, the demand for effective deepfake detectors has become essential. We are inspired by the fact that normal pulse rhythms present in real-face video can be decreased or even completely interrupted in a deepfake video; thus, we have introduced a new deepfake detection approach based on remote heart rate estimation using the 3D Central Difference Convolution Attention Network (CDCAN). Methods: Our proposed fake detector is mainly composed of a 3D CDCAN with an inverse attention mechanism and LSTM architecture. It utilizes 3D central difference convolution to enhance the spatiotemporal representation, which can capture rich physiological-related temporal context by gathering the time difference information. The soft attention mechanism is to focus on the skin region of interest, while the inverse attention mechanism is to further denoise rPPG signals. Results: The performance of our approach is evaluated on the two latest Celeb-DF and DFDC datasets, for which the experiment results show that our proposed approach achieves an accuracy of 99.5% and 97.4%, respectively. Conclusion: Our approach outperforms the state-of-art methods and proves the effectiveness of our DeepFake detector.

Remote Heart Rate Estimation Using Attention-targeted Self-Supervised Learning Methods

Heart rate measurement is a crucial factor for assessing the overall health status of an individual. Abnormal heart rates, whether lower or higher than baseline, can indicate potential pathological or physiological abnormalities. As a result, it is necessary to have reliable technology for monitoring heart rates in various fields, including medicine, biotechnology, and healthcare. With recent advancements in deep learning research, it is now possible to monitor heart rate conveniently and hygienically without specialized equipment, using facial video photo volume measurement. This new technology employs a deep learning-based video analysis method that requires a large data set to achieve high performance. However, collecting and labeling a vast amount of data is often impractical and costly. Therefore, researchers have been searching for alternative ways to achieve high performance with smaller datasets. This paper proposes a novel self-supervised learning approach suitable to the face video process. Our proposed method can effectively acquire a deep latent expression from a face image sequence and apply it to a target task through transfer learning. Using this method, we aim to improve the remote heart rate estimation performance in a limited-size dataset. Our proposed method is specialized for facial image sequences and focuses on the color change of the face to achieve high performance in existing attention-based deep learning models. The proposed self-supervised learning method has several advantages. First, it can learn useful features from unlabeled data, reducing the reliance on annotated datasets. Second, it can help overcome the problem of insufficient labeled data in specific domains, such as medical image analysis. Third, the proposed method can improve the performance of the target task using pre-trained models on different datasets. Finally, our approach improves the remote heart rate estimation performance by extracting useful features from facial images.

Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation

In recent years, the physiological measurement based on remote photoplethysmography has attracted wide attention, especially since the epidemic of COVID-19. Many researchers paid great efforts to improve the robustness of illumination and motion variation. Most of the existing methods divided the ROIs into many sub-regions and extracted the heart rate separately, while ignoring the fact that the heart rates from different sub-regions are consistent. To address this problem, in this work, we propose a structural sparse representation method to reconstruct the pulse signals (SSR2RPS) from different sub-regions and estimate the heart rate. The structural sparse representation (SSR) method considers that the chrominance signals from different sub-regions should have a similar sparse representation on the combined dictionary. Specifically, we firstly eliminate the signal deviation trend using the adaptive iteratively re-weighted penalized least squares (Airpls) for each sub-region. Then, we conduct the sparse representation on the combined dictionary, which is constructed considering the pulsatility and periodicity of the heart rate. Finally, we obtain the reconstructed pulse signals from different sub-regions and estimate the heart rate with a power spectrum analysis. The experimental results on the public UBFC and COHFACE datasets demonstrate the significant improvement for the accuracy of the heart rate estimation under realistic conditions.

Image based control of smart workout systems

This research is dedicated to develop a forward-looking image-based physiological monitoring system that could be incorporated into general fitness facilities to realize feedback control cooperated with the non-contact monitoring sensor of the subject's physiological status. We propose a fitness control architecture applied to the fitness workout equipment in an ergonomic manner. The control scheme involves a webcam and a fuzzy control system which inferences control strategy according to the subject’s physical status to track the object’s intention of exercising intensity. The integrated feedback information includes heart rate (HR), HR variability, and exercising torque. Under the strenuous excising condition, the proposed algorithm with the adaptive filtering feature in remote HR estimation accuracy shows 14.4 ∼ 17.3% improvement over the traditional approaches. Moreover, a class of smart fitness sports modes is also proposed to incorporate with the fuzzy control scheme. Finally, real-world experimental results are presented to demonstrate the proposed design.

CoSTHR: A Heart Rate Estimating Network With Adaptive Color Space Transformation

Heart rate (HR) is one of the most important physiological signals of a person. Remote HR estimation derives subtle color variations from one’s facial video captured by a camera. The periodic signal caused by heart activity is then extracted from the color variations, resulting in heart rate. The advantages of remote methods show in their contactlessness, comfortableness, and impalpability. Due to the very subtle color changes of the face, remote methods are quite sensitive to illumination variations and motion artifacts. In current studies, researchers have found that the results achieved from experiments conducted on different color spaces vary significantly, suggesting that the choice of color space has an important impact on HR estimation. However, it is difficult to propose an optimum color space that is appropriate for the HR estimation task. Additionally, there is a lack of specialized training strategies for color space projection tasks as well as combinations with the state-of-the-art mechanisms. In this article, we propose a HR estimating network with adaptive color space transformation, abbreviated as CoSTHR, to address these challenges. The network uses color space transformation (CoST) layers to learn the optimum color space for HR estimation task, based on which the attention convolutional neural network can focus on the informative channels and extract the HR. Specialized training strategies are further introduced to enhance the model performance, yielding more accurate results. Experiments on dataset VIPL-HR show that the proposed CoSTHR achieves better results than the state-of-the-art algorithms.

Deep-HR: Fast heart rate estimation from face video under realistic conditions

This paper presents a novel method for remote heart rate (HR) estimation. Recent studies have proved that blood pumping by the heart is highly correlated to the intense color of face pixels, and surprisingly can be utilized for remote HR estimation. Researchers successfully proposed several methods for this task, but making it work in realistic situations is still a challenging problem in computer vision community. Furthermore, learning to solve such a complex task on a dataset with very limited annotated samples is not reasonable. Consequently, researchers do not prefer to use the deep learning approaches for this problem. In this paper, we propose a simple yet efficient approach to benefit the advantages of the Deep Neural Network (DNN) by simplifying HR estimation from a complex task to learning from very correlated representation to HR. Inspired by previous work, we learn a component called Front-End (FE) to provide a discriminative representation of face videos, afterward a light deep regression auto-encoder as Back-End (BE) is learned to map the FE representation to HR. Regression task on the informative representation is simple and could be learned efficiently on limited training samples. Beside of this, to be more accurate and work well on low-quality videos, two deep encoder–decoder networks are trained to refine the output of FE. We also introduce a challenging dataset (HR-D) to show that our method can efficiently work in realistic conditions. Experimental results on HR-D and MAHNOB datasets confirm that our method could run as a real-time method and estimate the average HR better than state-of-the-art ones.

Spatial-Spectral-Temporal Fusion for Remote Heart Rate Estimation

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos to improve the robustness of camera-based remote HR estimation against illumination fluctuations and head motions. The key to robust HR estimation is constructing the histogram of HRs for a considered time window by voting candidate HRs that are estimated using different spatial face patches, spectral modalities (i.e., RGB and NIR), and temporal short-time sub-windows. The histogram voting is performed only for the candidate HRs that pass through a reliability check of HR estimation. The final HR estimate for the considered time window is then obtained by detecting the most frequently voted HR bin and performing parabola fitting using its neighboring bins. By spatially, spectrally, and temporally fusing the candidate HRs for majority voting, our method can automatically exploit suitable video sub-regions less affected by illumination fluctuations and head motions to enable robust HR estimation. Through the experiments on 168 RGB-NIR video recordings, we demonstrate that our fusion-based method achieves improved HR estimation accuracy compared with existing methods.

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation.

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.