rPPG Signal Research Articles

AVENUE: A novel deep fake detection method based on temporal convolution network & rPPG information

In Deep Learning (DL), an adversary creates Deepfakes by manipulating facial features to fool someone. The Deepfakes pose a security threat to anyone's privacy and a primary concern for our society. It can be detected by utilizing the texture and physiological properties of the face, like eye and lip movements; however, such methods are incompetent when Deepfakes are created using recent generative adversarial networks (GAN). Alternatively, remote Photoplethysmography (rPPG) information can be used for Deepfake detection because GANs neglect human physiological information for Deepfake generation. Such detection can be inaccurate when rPPG signals are affected by the noises induced by facial deformation and illumination variations. Furthermore, the exiting Deepfake detections are usually performed using sequential models, and such models fail to process the long sequence of temporal information. These issues are mitigated by our proposed method AVENUE , that is, \(A\) no \(V\) el d \(E\) epfake detectio \(N\) method based on temporal convol \(U\) tion n \(E\) twork & rPPG information. For mitigating the noise issues in the rPPG signals, the proposed method detects and employs relatively stable clips of the input video for Deepfake detection. The stable clips are those clips that are least affected by facial deformations. Also, we use a modified Temporal convolution network to model the long sequence of Deepfake information rather than the sequential architectures. We performed the experimental result on publically available datasets of Deepfake videos. It demonstrates that our proposed method performs better than the existing rPPG-based Deepfake detection methods.

An image enhancement based method for improving rPPG extraction under low-light illumination

Remote photoplethysmography (rPPG) has gained significant attention as a non-invasive approach for measuring human vital signs from videos. However, the reliance on capturing the reflection of ambient lighting causes it to be susceptible to the adverse effects of low illumination levels, leading to deterioration in the signal quality. The preliminary study introduces a novel methodology for enhancing rPPG signal extraction in low-light conditions through the integration of an Image Enhancement Model (IEM) inspired by Retinex theory, which significantly improved signal quality by preprocessing video frames to better capture the subtle changes in facial blood flow. Recognizing the challenge posed by the generalization capacity over different deep-learning models and unseen examples from various datasets, this study further evaluated the efficacy of the IEM + rPPG extraction model across multiple datasets (UBFC-rPPG, BH-rPPG, MMPD-rPPG, and VIPL-HR) by integrating IEM with existing deep-learning based rPPG extraction models including DeepPhys, PhysNet, and PhysFormer and traditional POS rPPG extraction method. Our experiments demonstrate a consistent improvement for all the methods in heart rate estimation accuracy across all datasets, underscoring the IEM’s adaptability and effectiveness. The paper also explores the application of our method to traditional rPPG extraction techniques, further validating its potential for broader usage. Through comprehensive analysis, this research not only confirms the impact of lighting conditions on rPPG signal quality but also provides a robust solution for more reliable non-invasive vital sign monitoring in diverse environments.

Facial Video-Based Non-Contact Stress Recognition Utilizing Multi-Task Learning With Peak Attention.

Negative emotional states, such as anxiety and depression, pose significant challenges in contemporary society, often stemming from the stress encountered in daily activities. Stress (state or level) recognition is a crucial prerequisite for effective stress management and intervention. Presently, wearable devices have been employed to capture physiological signals and analyze stress states. However, their constant skin contact can lead to discomfort and disturbance during prolonged monitoring. In this paper, a peak attention-based multitasking framework is presented for non-contact stress recognition. The framework extracts rPPG signals from RGB facial videos, utilizing them as inputs for a novel multi-task attentional convolutional neural network for stress recognition (MTASR). It incorporates peak detection and HR estimation as auxiliary tasks to facilitate stress recognition. By leveraging multi-task learning, MTASR can utilize information related to stress physiological responses, thereby enhancing feature extraction efficiency. For stress recognition, two binary classification tasks are applied: stress state recognition and stress level recognition. The model is validated on the UBFC-Phys public dataset and demonstrates an accuracy of 94.33% for stress state recognition and 83.83% for stress level recognition. The proposed method outperforms the dataset's baseline methods and other competing approaches.

What Remote PPG Oximetry Tells Us about Pulsatile Volume?

While pulse oximetry using remote photoplethysmography (rPPG) is used in medicine and consumer health, sound theoretical foundations for this methodology are not established. Similarly to traditional pulse oximetry, rPPG oximetry uses two wavelengths to calculate the tissue oxygenation using the so-called ratio-of-ratios, R. However, the relationship between R and tissue oxygenation has not been derived analytically. As such, rPPG oximetry relies mostly on empirical methods. This article aimed to build theoretical foundations for pulse oximetry in rPPG geometry. Using the perturbation approach in diffuse approximation for light propagation in tissues, we obtained an explicit expression of the AC/DC ratio for the rPPG signal. Based on this ratio, the explicit expression for "ratio-of-ratios" was obtained. We have simulated the dependence of "ratio-of-ratios" on arterial blood saturation across a wide range (SaO2 = 70-100%) for several commonly used R/IR light sources (660/780, 660/840, 660/880, and 660/940 nm) and found that the obtained relationship can be modeled by linear functions with an extremely good fit (R2 = 0.98-0.99) for all considered R/IR pairs. Moreover, the location of the pulsatile volume can be extracted from rPPG data. From experimental data, we found that the depth of blood pulsations in the human forehead can be estimated as 0.6 mm on the arterial side, which points to the papillary dermis/subpapillary vascular plexus origin of the pulsatile volume.

ST-Phys: Unsupervised Spatio-Temporal Contrastive Remote Physiological Measurement.

Remote photoplethysmography (rPPG) is a non-contact method that employs facial videos for measuring physiological parameters. Existing rPPG methods have achieved remarkable performance. However, the success mainly profits from supervised learning over massive labeled data. On the other hand, existing unsupervised rPPG methods fail to fully utilize spatio-temporal features and encounter challenges in low-light or noise environments. To address these problems, we propose an unsupervised contrast learning approach, ST-Phys. We incorporate a low-light enhancement module, a temporal dilated module, and a spatial enhanced module to better deal with long-term dependencies under the random low-light conditions. In addition, we design a circular margin loss, wherein rPPG signals originating from identical videos are attracted, while those from distinct videos are repelled. Our method is assessed on six openly accessible datasets, including RGB and NIR videos. Extensive experiments reveal the superior performance of our proposed ST-Phys over state-of-the-art unsupervised rPPG methods. Moreover, it offers advantages in parameter reduction and noise robustness.

Contrast-Phys+: Unsupervised and Weakly-Supervised Video-Based Remote Physiological Measurement via Spatiotemporal Contrast.

Video-based remote physiological measurement utilizes facial videos to measure the blood volume change signal, which is also called remote photoplethysmography (rPPG). Supervised methods for rPPG measurements have been shown to achieve good performance. However, the drawback of these methods is that they require facial videos with ground truth (GT) physiological signals, which are often costly and difficult to obtain. In this paper, we propose Contrast-Phys+, a method that can be trained in both unsupervised and weakly-supervised settings. We employ a 3DCNN model to generate multiple spatiotemporal rPPG signals and incorporate prior knowledge of rPPG into a contrastive loss function. We further incorporate the GT signals into contrastive learning to adapt to partial or misaligned labels. The contrastive loss encourages rPPG/GT signals from the same video to be grouped together, while pushing those from different videos apart. We evaluate our methods on five publicly available datasets that include both RGB and Near-infrared videos. Contrast-Phys+ outperforms the state-of-the-art supervised methods, even when using partially available or misaligned GT signals, or no labels at all. Additionally, we highlight the advantages of our methods in terms of computational efficiency, noise robustness, and generalization.

End-to-end multimodal emotion recognition based on facial expressions and remote photoplethysmography signals.

Emotion is a complex physiological phenomenon, and a single modality may be insufficient for accurately determining human emotional states. This paper proposes an end-to-end multimodal emotion recognition method based on facial expressions and non-contact physiological signals. Facial expression features and remote photoplethysmography (rPPG) signals are extracted from facial video data, and a transformer-based cross-modal attention mechanism (TCMA) is used to learn the correlation between the two modalities. The results show that the accuracy of emotion recognition can be slightly improved by combining facial expressions with accurate rPPG signals. The performance is further improved with the use of TCMA, for which the binary classification accuracy of valence and arousal is 91.11% and 90.00%, respectively. Additionally, when experiments are conducted using the whole dataset, an increased accuracy of 7.31% and 4.23% for the binary classification of valence and arousal, and an improved accuracy of 5.36% for the four classifications of valence-arousal are achieved when TCMA is used in modal fusion, compared to using only facial expression modality, which fully demonstrates the effectiveness and robustness of TCMA. This method makes it possible to realize multimodal emotion recognition of facial expressions and contactless physiological signals in reality.

Optimal signal quality index for remote photoplethysmogram sensing

Remote photoplethysmography (rPPG) enables non-invasive monitoring of circulatory signals using mobile devices, a crucial advancement in biosensing. Despite its potential, ensuring signal quality amidst noise and artifacts remains a significant challenge, particularly in healthcare applications. Addressing this, our study focuses on a singular signal quality index (SQI) for rPPG, aimed at simplifying high-quality video capture for heart rate detection and cardiac assessment. We introduce a practical threshold for this SQI, specifically the signal-to-noise ratio index (NSQI), optimized for straightforward implementation on portable devices for real-time video analysis. Employing (NSQI < 0.293) as our threshold, our methodology successfully identifies high-quality cardiac information in video frames, effectively mitigating the influence of noise and artifacts. Validated on publicly available datasets with advanced machine learning algorithms and leave-one-out cross-validation, our approach significantly reduces computational complexity. This innovation not only enhances efficiency in health monitoring applications but also offers a pragmatic solution for remote biosensing. Our findings constitute a notable advancement in rPPG signal quality assessment, marking a critical step forward in the development of remote cardiac monitoring technologies with extensive healthcare implications.

A machine learning-based approach for constructing remote photoplethysmogram signals from video cameras

BackgroundAdvancements in health monitoring technologies are increasingly relying on capturing heart signals from video, a method known as remote photoplethysmography (rPPG). This study aims to enhance the accuracy of rPPG signals using a novel computer technique.MethodsWe developed a machine-learning model to improve the clarity and accuracy of rPPG signals by comparing them with traditional photoplethysmogram (PPG) signals from sensors. The model was evaluated across various datasets and under different conditions, such as rest and movement. Evaluation metrics, including dynamic time warping (to assess timing alignment between rPPG and PPG) and correlation coefficients (to measure the linear association between rPPG and PPG), provided a robust framework for validating the effectiveness of our model in capturing and replicating physiological signals from videos accurately.ResultsOur method showed significant improvements in the accuracy of heart signals captured from video, as evidenced by dynamic time warping and correlation coefficients. The model performed exceptionally well, demonstrating its effectiveness in achieving accuracy comparable to direct-contact heart signal measurements.ConclusionsThis study introduces a novel and effective machine-learning approach for improving the detection of heart signals from video. The results demonstrate the flexibility of our method across various scenarios and its potential to enhance the accuracy of health monitoring applications, making it a promising tool for remote healthcare.

General ROI Selection Approach in rPPG for Accurate Heart Rate Estimation

Selecting the optimal region of interest (ROI) in remote photoplethysmography (rPPG) is crucial for capturing desired pulsatile information accurately. We propose a novel method to identify the best subject-dependent region by employing variable-size segmentation to meet minimum accuracy criteria for heart rate estimation. By selecting the intersection of these segments, we determine the subject-dependent optimal region. Finally, through geometrical transformations and the amalgamation of all subject-dependent regions, we derive the ultimate subject-independent ROI. Our findings indicate that the optimal region for extracting rPPG signals is an area on the forehead, encompassing the mid-nasal ridge within the supratrochlear vein passage. Additionally, heart rate estimation utilizing this specified region achieved accuracy rates above 98% across all cases, demonstrating the effectiveness of this region for rPPG estimation. Through our research, we aim to contribute to the ongoing development of precise and robust physiological measurements using rPPG technology.

Exploring neural motion transfer for unsupervised remote physiological measurement: A practicality study

Remote Photoplethysmography (rPPG) is a method to measure cardiac activity without the need for any contact-based sensors, garnering attention due to its non-invasive and convenient nature. Addressing the issue of substantial missing data in large-scale datasets, current research is inclined towards unsupervised learning methods. However, prevailing unsupervised learning methods often prioritize inter-sample comparative learning while neglecting individual sample features, severely impacting performance and generalizability. Meanwhile, body movement is one of the most important influencing factors when attempting to extract the rPPG signal from videos. Hence, we embarked on the inaugural exploration of integrating Neural Motion Transfer into unsupervised learning methods for remote physiological measurements, aiming to enrich the inherent motion characteristics and diversity of samples. Employing Neural Motion Transfer, we synthesized videos encompassing diverse movements as a means of data augmentation. Through an in-depth comparison with other synthetic video methods, we extensively analyzed the physiological information within motion-enhanced synthetic videos. We investigated the applicability of these motion-enhanced synthetic videos to unsupervised learning methods and various associated impacts. We conducted extensive experiments on four benchmark datasets. After utilizing motion-enhanced synthetic videos to aid unsupervised training, our approach achieved an improvement of approximately 52% in intra-dataset testing and up to 97% in cross-dataset testing compared to the baseline unsupervised method, achieving the state-of-the-art (SOTA) performance among current unsupervised methods. Our research underscores the importance of neural motion networks in enhancing unsupervised learning methods and highlights the significant potential of synthetic videos.

A self-supervised learning network for remote heart rate measurement

The rPPG technology is a facial video-based non-contact physiological parameter detection technology, which has been widely used in the measurement of heart rate, respiration, blood oxygen and other physiological parameters. Most of the current research hotspots in this field are based on deep learning methods, which typically necessitate a significant amount of data samples for training the neural network to obtain accurate measurement results. However, the high cost required to label the data samples limits the diffusion and application of the technique. To address the above problems, we propose a new self-supervised aimed at acquiring the ability to estimate rPPG signals from facial videos, eliminating the need for labeled data. The framework expands the unlabeled video samples into multiple positive/negative samples and uses contrast learning to obtain an rPPG signal estimation network, which outputs the corresponding rPPG signals of face videos. To promote the convergence of network training, a new frequency loss function is designed. This function can effectively shorten the distance between the frequencies of similar sample signals and push the distance between the frequencies of different sample signals far away, so as to enhance the frequency consistency between sample signals and make the model easier to learn the differences between different sample pairs. Our method is evaluated on four standard rPPG datasets, the experimental results show that the accuracy is close to the current best supervised method, and is superior to the previous self-supervised method without using any labeled data.

Learning Spatio-Temporal Pulse Representation With Global-Local Interaction and Supervision for Remote Prediction of Heart Rate.

Recent studies have demonstrated the benefit of extracting and fusing pulse signals from multi-scale region-of-interests (ROIs). However, these methods suffer from heavy computational load. This paper aims to effectively utilize multi-scale rPPG features with a more compact architecture. Inspired by recent research works exploring two-path architecture that leverages global and local information with bidirectional bridge in between. This paper designs a novel architecture Global-Local Interaction and Supervision Network (GLISNet), which uses a local path to learn representations in the original scale and a global path to learn representations in the other scale capturing multi-scale information. A light-weight rPPG signal generation block is attached to the output of each path that maps the pulse representation to the pulse output. A hybrid loss function is utilized enabling the local and global representations to learn directly from the training data. Extensive experiments are conducted on two publicly available datasets, and results demonstrate that GLISNet achieves superior performance in terms of signal-to-noise ratio (SNR), mean absolute error (MAE), and root mean squared error (RMSE). In terms of SNR, GLISNet has an increase of 4.41% compared with the second best algorithm PhysNet on PURE dataset. The MAE has a decrease of 13.16% compared with the second best algorithm DeeprPPG on UBFC-rPPG dataset. The RMSE has a decrease of 26.29% compared with the second best algorithm PhysNet on UBFC-rPPG dataset. Experiments on MIHR dataset demonstrates the robustness of GLISNet under low-light environment.

Contactless Blood Pressure Measurement Via Remote Photoplethysmography With Synthetic Data Generation Using Generative Adversarial Networks.

Remote photoplethysmography (rPPG) has been used to measure vital signs such as heart rate, heart rate variability, blood pressure (BP), and blood oxygen. Recent studies adopt features developed with photoplethysmography (PPG) to achieve contactless BP measurement via rPPG. These features can be classified into two groups: time or phase differences from multiple signals, or waveform feature analysis from a single signal. Here we devise a solution to extract the time difference information from the rPPG signal captured at 30 FPS. We also propose a deep learning model architecture to estimate BP from the extracted features. To prevent overfitting and compensate for the lack of data, we leverage a multi-model design and generate synthetic data. We also use subject information related to BP to assist in model learning. For real-world usage, the subject information is replaced with values estimated from face images, with performance that is still better than the state-of-the-art. To our best knowledge, the improvements can be achieved because of: 1) the model selection with estimated subject information, 2) replacing the estimated subject information with the real one, 3) the InfoGAN assistance training (synthetic data generation), and 4) the time difference features as model input. To evaluate the performance of the proposed method, we conduct a series of experiments, including dynamic BP measurement for many single subjects and nighttime BP measurement with infrared lighting. Our approach reduces the MAE from 15.49 to 8.78 mmHg for systolic blood pressure (SBP) and 10.56 to 6.16 mmHg for diastolic blood pressure(DBP) on a self-constructed rPPG dataset. On the Taipei Veterans General Hospital(TVGH) dataset for nighttime applications, the MAE is reduced from 21.58 to 11.12 mmHg for SBP and 9.74 to 7.59 mmHg for DBP, with improvement ratios of 48.47% and 22.07% respectively.

Standardized rPPG signal generation based on generative adversarial networks

Standardized rPPG signal generation based on generative adversarial networks

Evaluating RGB channels in remote photoplethysmography: a comparative study with contact-based PPG.

Remote photoplethysmography (rPPG) provides a non-contact method for measuring blood volume changes. In this study, we compared rPPG signals obtained from video cameras with traditional contact-based photoplethysmography (cPPG) to assess the effectiveness of different RGB channels in cardiac signal extraction. Our objective was to determine the most effective RGB channel for detecting blood volume changes and estimating heart rate. We employed dynamic time warping, Pearson's correlation coefficient, root-mean-square error, and Beats-per-minute Difference to evaluate the performance of each RGB channel relative to cPPG. The results revealed that the green channel was superior, outperforming the blue and red channels in detecting volumetric changes and accurately estimating heart rate across various activities. We also observed that the reliability of RGB signals varied based on recording conditions and subject activity. This finding underscores the importance of understanding the performance nuances of RGB inputs, crucial for constructing rPPG signals in algorithms. Our study is significant in advancing rPPG research, offering insights that could benefit clinical applications by improving non-contact methods for blood volume assessment.

Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals

Depression stands as one of the most widespread psychological disorders and has garnered increasing attention. Currently, how to effectively achieve automatic multimodal depression detection for assisting doctors in early diagnosis of depression, has become an important and challenging issue. To address this issue, this work proposes Transformer-based feature enhancement networks for multimodal depression detection. The proposed method effectively integrates three modalities including video, audio and remote photoplethysmographic (rPPG) signals for multimodal depression detection, in which the rPPG modality is introduced as an additional modality for enhancing the effectiveness of multimodal depression detection. The proposed method consists of three key steps: multimodal feature extraction for video, audio and rPPG modalities, Transformer-based multimodal feature enhancement (TMFE), and graph fusion networks (GFN) based multimodal fusion and depression prediction. More specially, in the stage of multimodal feature extraction, for video and audio modalities we employ deep convolutional neural networks (CNN) to extract the corresponding high-level video and audio features, respectively. For rPPG modality, we adopt a short-time end-to-end rPPG estimation framework to extract the rPPG signal values. The TMFE module stacks multiple Transformers such as the inter-modal, intra-modal, and tri-modal Transformers to jointly capture the dynamics and relationships within and between modalities for each time-step of input sequences. The GFN module is designed to effectively fuse the obtained feature representations from different modalities while maintaining the interactions between them simultaneously. Finally, the obtained shared feature representations of all modalities are fed into a multilayer perceptrons (MLP) network to implement final depression detection tasks. Extensive experiments are conducted on two public datasets such as AVEC2013 and AVEC2014, and experimental results demonstrate the validity of the proposed method on depression detection tasks.

LGI-rPPG-Net: A shallow encoder-decoder model for rPPG signal estimation from facial video streams

A method to accurately estimate physiological signals from video streams at a minimal cost is invaluable. The importance of such a technique in pre-clinical health monitoring cannot be understated. Remote photoplethysmography (rPPG) can be used as a substitute for finger photoplethysmography (PPG) when such sensors are not recommended, such as for burn victims, premature babies, and patients with sensitive skin. Good quality rPPG signal that is highly correlated to finger PPG can be used to estimate many vital health signs. In this work, a shallow encoder-decoder architecture, LGI-rPPG-Net is proposed. The proposed model aims to produce highly correlated rPPG signals which can be substituted for finger PPG. In the reconstruction of rPPG, the model achieved a very good Pearson’s Correlation Coefficient (PCC), Root Mean Squared Error (RMSE), and dynamic time warping distance of 0.862, 0.148, and 0.699, respectively. This highly correlated rPPG was compared to finger PPG by calculating heart rate from rPPG and finger PPG. The model achieved a PCC of 0.984 and RMSE, and MAE of 2.91, 1.51 beats per minute (BPM), respectively. LGI-rPPG-Net model with video streaming to predict rPPG can thus be used as a replacement for finger PPG where in-contact collection is not feasible.

Face2PPG: An Unsupervised Pipeline for Blood Volume Pulse Extraction from Faces.

Photoplethysmography (PPG) signals have become a key technology in many fields, such as medicine, well-being, or sports. Our work proposes a set of pipelines to extract remote PPG signals (rPPG) from the face robustly, reliably, and configurable. We identify and evaluate the possible choices in the critical steps of unsupervised rPPG methodologies. We assess a state-of-the-art processing pipeline in six different datasets, incorporating important corrections in the methodology that ensure reproducible and fair comparisons. In addition, we extend the pipeline by proposing three novel ideas; 1) a new method to stabilize the detected face based on a rigid mesh normalization; 2) a new method to dynamically select the different regions in the face that provide the best raw signals, and 3) a new RGB to rPPG transformation method, called Orthogonal Matrix Image Transformation (OMIT) based on QR decomposition, that increases robustness against compression artifacts. We show that all three changes introduce noticeable improvements in retrieving rPPG signals from faces, obtaining state-of-the-art results compared with unsupervised, non-learning-based methodologies and, in some databases, very close to supervised, learning-based methods. We perform a comparative study to quantify the contribution of each proposed idea. In addition, we depict a series of observations that could help in future implementations.

Depression Recognition Using Remote Photoplethysmography From Facial Videos

Depression is a mental illness that may be harmful to an individual's health. The detection of mental health disorders in the early stages and a precise diagnosis are critical to avoid social, physiological, or psychological side effects. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) response. Although typically, HRV features are calculated from biosignals obtained with contact-based sensors such as wearables, we propose instead a novel scheme that directly extracts them from facial videos, just based on visual information, removing the need for any contact-based device. Our solution is based on a pipeline that is able to extract complete remote photoplethysmography signals (rPPG) in a fully unsupervised manner. We use these rPPG signals to calculate over 60 statistical, geometrical, and physiological features that are further used to train several machine learning regressors to recognize different levels of depression. Experiments on two benchmark datasets indicate that this approach offers comparable results to other audiovisual modalities based on voice or facial expression, potentially complementing them. In addition, the results achieved for the proposed method show promising and solid performance that outperforms hand-engineered methods and is comparable to deep learning-based approaches.