Face Video Research Articles

Driver Multi-task Emotion Recognition Network Based on Multi-modal Facial Video Analysis

Driver emotion recognition is crucial for enhancing the safety and user experience in driving scenarios. However, current emotion recognition methods often rely solely on a single modality and a single-task setup, leading to suboptimal performance in driving scenarios. To address this, this paper proposes a driver multitask emotion recognition method based on multimodal facial video analysis (MER-MFVA). This method extracts facial expression features and remote photoplethysmography (rPPG) signals from driver facial videos. Facial expression features include facial action units and eye movement information, representing the driver's external characteristics. rPPG information, representing the driver's internal characteristics, is enhanced through a designed dual-path Transformer network and an introduced focus module. We also propose a cross-modal mutual attention computation mechanism to effectively fuse multimodal features by calculating mutual attention between facial expression features and rPPG information. In the final task output, we employ a multitask learning mechanism, setting discrete emotion recognition as the primary task and emotion valence recognition, emotion arousal recognition, and the previous rPPG information extraction as auxiliary tasks to facilitate effective information sharing across different tasks. Experimental results on the established driver emotion dataset demonstrate that our proposed method significantly improves driver emotion recognition performance, achieving an accuracy of 86.98% and an F1 score of 85.83% in the primary task. This validates the effectiveness of the proposed approach.

Improved Remote Photoplethysmography Using Machine Learning-Based Filter Bank

Remote photoplethysmography (rPPG) is a non-contact technology that monitors heart activity by detecting subtle color changes within the facial blood vessels. It provides an unconstrained and unconscious approach that can be widely applied to health monitoring systems. In recent years, research has been actively conducted to improve rPPG signals and to extract significant information from facial videos. However, rPPG can be vulnerable to degradation due to changes in the illumination and motion of a subject, and overcoming these challenges remains difficult. In this study, we propose a machine learning-based filter bank (MLFB) noise reduction algorithm to improve the quality of rPPG signals. The MLFB algorithm determines the optimal spectral band for extracting information on cardiovascular activity and reconstructing an rPPG signal using a support vector machine. The proposed approach was validated with an open dataset, achieving a 35.5% (i.e., resulting in a mean absolute error of 2.5 beats per minute) higher accuracy than those of conventional methods. The proposed algorithm can be integrated into various rPPG algorithms for the pre-processing of RGB signals. Moreover, its computational efficiency is expected to enable straightforward implementation in system development, making it broadly applicable across the healthcare field.

MYFED: a dataset of affective face videos for investigation of emotional facial dynamics as a soft biometric for person identification

MYFED: a dataset of affective face videos for investigation of emotional facial dynamics as a soft biometric for person identification

Frontal EEG correlation based human emotion identification and classification.

Humans express their feelings and intentions of their actions or communication through emotions. Recent advancements in technology involve machines in human communication in day-to-day life. Thus, understanding of human emotions by machines will be very helpful in assisting the user in a far better way. Various physiological and non-physiological signals can be used to make the machines to recognize the emotion of a person. The identification of emotional content in the signals is crucial to understand emotion and the machines act with emotional intelligence at appropriate times, thus providing a better human machine interaction with emotion identification system and mental health monitoring for psychiatric patients. This work includes the creation of an emotion EEG dataset, the development of an algorithm for identifying the emotion elicitation segments in the EEG signal, and the classification of emotions from EEG signals. The EEG signals are divided into 3s segments, and the segments with emotional content are selected based on the decrease in correlation between the frontal electrodes. The selected segments are validated with the facial expressions of the subjects in the appropriate time segments of the face video. EEGNet is used to classify the emotion from the EEG signal. The classification accuracy with the selected emotional EEG segments is higher compared to the accuracy using all the EEG segments. In subject-specific classification, an average accuracy of 80.87% is obtained from the network trained with selected EEG segments, and 70.5% is obtained from training with all EEG segments. In subject-independent classification, the accuracy of classification is 67% and 63.8% with and without segment selection, respectively. The proposed method of selection of EEG segments is validated using the DEAP dataset, and classification accuracies and F1-scores of subject dependent and subject-independent methods are presented.

Heart Rate Estimation Algorithm Integrating Long and Short-Term Temporal Features

Non-contact heart rate monitoring from facial videos utilizing remote photoplethysmography (rPPG) has gained significant traction in remote health monitoring. Given that rPPG captures the dynamic blood flow within the human body and constitutes a time-series signal characterized by periodic properties, this study introduced a three-dimensional convolutional neural network (3D CNN) designed to simultaneously address long-term periodic and short-term temporal characteristics for effective rPPG signal extraction. Firstly, differential operations are employed to preprocess video data, enhancing the face’s dynamic features. Secondly, building upon the 3D CNN framework, multi-scale dilated convolutions and self-attention mechanisms were integrated to enhance the model’s temporal modeling capabilities further. Finally, interpolation techniques are applied to refine the heart rate calculation methodology. The experiments conducted on the UBFC-rPPG dataset indicate that, compared with the existing optimal algorithm, the average absolute error (MAE) and the root mean square error (RMSE) realized significant enhancements of approximately 28% and 35%. Additionally, through comprehensive analyses such as cross-dataset experiments and complexity analyses, the validity and stability of the proposed algorithm in the task of heart rate estimation were manifested.

MDAR: A Multiscale Features-Based Network for Remotely Measuring Human Heart Rate Utilizing Dual-Branch Architecture and Alternating Frame Shifts in Facial Videos.

Remote photoplethysmography (rPPG) refers to a non-contact technique that measures heart rate through analyzing the subtle signal changes of facial blood flow captured by video sensors. It is widely used in contactless medical monitoring, remote health management, and activity monitoring, providing a more convenient and non-invasive way to monitor heart health. However, factors such as ambient light variations, facial movements, and differences in light absorption and reflection pose challenges to deep learning-based methods. To solve these difficulties, we put forward a measurement network of heart rate based on multiscale features. In this study, we designed and implemented a dual-branch signal processing framework that combines static and dynamic features, proposing a novel and efficient method for feature fusion, enhancing the robustness and reliability of the signal. Furthermore, we proposed an alternate time-shift module to enhance the model's temporal depth. To integrate the features extracted at different scales, we utilized a multiscale feature fusion method, enabling the model to accurately capture subtle changes in blood flow. We conducted cross-validation on three public datasets: UBFC-rPPG, PURE, and MMPD. The results demonstrate that MDAR not only ensures fast inference speed but also significantly improves performance. The two main indicators, MAE and MAPE, achieved improvements of at least 30.6% and 30.2%, respectively, surpassing state-of-the-art methods. These conclusions highlight the potential advantages of MDAR for practical applications.

GCS-YOLOv8: A Lightweight Face Extractor to Assist Deepfake Detection.

To address the issues of target feature blurring and increased false detections caused by high compression rates in deepfake videos, as well as the high computational resource requirements of existing face extractors, we propose a lightweight face extractor to assist deepfake detection, GCS-YOLOv8. Firstly, we employ the HGStem module for initial downsampling to address the issue of false detections of small non-face objects in deepfake videos, thereby improving detection accuracy. Secondly, we introduce the C2f-GDConv module to mitigate the low-FLOPs pitfall while reducing the model's parameters, thereby lightening the network. Additionally, we add a new P6 large target detection layer to expand the receptive field and capture multi-scale features, solving the problem of detecting large-scale faces in low-compression deepfake videos. We also design a cross-scale feature fusion module called CCFG (CNN-based Cross-Scale Feature Fusion with GDConv), which integrates features from different scales to enhance the model's adaptability to scale variations while reducing network parameters, addressing the high computational resource requirements of traditional face extractors. Furthermore, we improve the detection head by utilizing group normalization and shared convolution, simplifying the process of face detection while maintaining detection performance. The training dataset was also refined by removing low-accuracy and low-resolution labels, which reduced the false detection rate. Experimental results demonstrate that, compared to YOLOv8, this face extractor achieves the AP of 0.942, 0.927, and 0.812 on the WiderFace dataset's Easy, Medium, and Hard subsets, representing improvements of 1.1%, 1.3%, and 3.7% respectively. The model's parameters and FLOPs are only 1.68 MB and 3.5 G, reflecting reductions of 44.2% and 56.8%, making it more effective and lightweight in extracting faces from deepfake videos.

Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos

Heart rate (HR) is an important indicator reflecting the overall physical and mental health of the human body, playing a crucial role in diagnosing cardiovascular and neurological diseases. Recent research has revealed that variations in the light absorption of human skin captured through facial video over the cardiac cycle, due to changes in blood volume, can be utilized for non-contact HR estimation. However, most existing methods rely on single-modal video sources (such as RGB or NIR), which often yield suboptimal results due to noise and the limitations of a single information source. To overcome these challenges, this paper proposes a multimodal information fusion architecture named the spatiotemporal sensitive network (SS-Net) for non-contact heart rate estimation. Firstly, spatiotemporal feature maps are utilized to extract physiological signals from RGB and NIR videos effectively. Next, a spatiotemporal sensitive (SS) module is introduced to extract useful physiological signal information from both RGB and NIR spatiotemporal maps. Finally, a multi-level spatiotemporal context fusion (MLSC) module is designed to fuse and complement information between the visible light and infrared modalities. Then, different levels of fused features are refined in task-specific branches to predict both remote photoplethysmography (rPPG) signals and heart rate (HR) signals. Experiments conducted on three datasets demonstrate that the proposed SS-Net achieves superior performance compared to existing methods.

Multimodal Information Fusion and Data Generation for Evaluation of Second Language Emotional Expression

This study aims to develop an emotion evaluation method for second language learners, utilizing multimodal information to comprehensively evaluate students’ emotional expressions. Addressing the limitations of existing emotion evaluation methods, which primarily focus on the acoustic features of speech (e.g., pronunciation, frequency, and rhythm) and often neglect the emotional expressions conveyed through voice and facial videos, this paper proposes an emotion evaluation method based on multimodal information. The method includes the following three main parts: (1) generating virtual data using a Large Language Model (LLM) and audio-driven facial video synthesis, as well as integrating the IEMOCAP dataset with self-recorded student videos and audios containing teacher ratings to construct a multimodal emotion evaluation dataset; (2) a graph convolution-based emotion feature encoding network to extract emotion features from multimodal information; and (3) an emotion evaluation network based on Kolmogorov–Arnold Networks (KAN) to compare students’ emotion features with standard synthetic data for precise evaluation. The emotion recognition method achieves an unweighted accuracy (UA) of 68.02% and an F1 score of 67.11% in experiments with the IEMOCAP dataset and TTS data. The emotion evaluation model, using the KAN network, outperforms the MLP network, with a mean squared error (MSE) of 0.811 compared to 0.943, providing a reliable tool for evaluating language learners’ emotional expressions.

The return of the uncanny: artificial intelligence and estranged futures

Although often considered to be a fault or a glitch in the system, the event of hallucination is integral to models of image-processing regularly employed by artificial intelligence (AI). Evoking an inhuman logic, the works in Trevor Paglen’s Adversarially Evolved Hallucinations series (2017–ongoing) render this hallucinatory space visible and open it up to investigation from within these systems. Produced by a generative adversarial network (GAN), the project reveals an uncanny domain of image production that raises significant misgivings about the routine use of generative AI (GenAI) in tasks associated with facial recognition and video surveillance. Through exploring the issues raised by Paglen in this work, the following essay enquires into questions related to machinic vision and AI more broadly. Through training neural networks – such as those employed in a GAN – to see the world for us, are we, to begin with, priming and instructing ourselves to see more like machines? What, furthermore, will happen when these inhuman, hallucinatory models of ‘seeing’ and processing the world supersede – in part or wholly – the sphere of human vision, not least in the adjacent fields of surveillance and contemporary models of automated warfare? Will these post-ocular models of seeing – which already determine, if not over-determine, our relationship to the world – become in turn entirely unaccountable, if not completely unfathomable, guidelines for defining normative and, more worryingly, non-normative behaviour patterns and everyday events. If, finally, AI models of image-processing replace ocular-centric ways of seeing, will these models have the capacity to further estrange, if not profoundly alienate, us from the world and our responsibility for the future impact and prejudicial potential of such images?

Deep fake Video Face Recognition Using Supervised Contrastive Learning for Scalability and Interpretability

Deep fake Video Face Recognition Using Supervised Contrastive Learning for Scalability and Interpretability

Respiratory Rate Estimation from Thermal Video Data Using Spatio-Temporal Deep Learning.

Thermal videos provide a privacy-preserving yet information-rich data source for remote health monitoring, especially for respiration rate (RR) estimation. This paper introduces an end-to-end deep learning approach to RR measurement using thermal video data. A detection transformer (DeTr) first finds the subject's facial region of interest in each thermal frame. A respiratory signal is estimated from a dynamically cropped thermal video using 3D convolutional neural networks and bi-directional long short-term memory stages. To account for the expected phase shift between the respiration measured using a respiratory effort belt vs. a facial video, a novel loss function based on negative maximum cross-correlation and absolute frequency peak difference was introduced. Thermal recordings from 22 subjects, with simultaneous gold standard respiratory effort measurements, were studied while sitting or standing, both with and without a face mask. The RR estimation results showed that our proposed method outperformed existing models, achieving an error of only 1.6 breaths per minute across the four conditions. The proposed method sets a new State-of-the-Art for RR estimation accuracy, while still permitting real-time RR estimation.

End-to-End Multimodal Emotion Recognition Based on Facial Expressions and Remote Photoplethysmography Signals.

Emotion is a complex physiological phenomenon, and a single modality may be insufficient for accurately determining human emotional states. This paper proposes an end-to-end multimodal emotion recognition method based on facial expressions and non-contact physiological signals. Facial expression features and remote photoplethysmography (rPPG) signals are extracted from facial video data, and a transformer-based cross-modal attention mechanism (TCMA) is used to learn the correlation between the two modalities. The results show that the accuracy of emotion recognition can be slightly improved by combining facial expressions with accurate rPPG signals. The performance is further improved with the use of TCMA, for which the binary classification accuracy of valence and arousal is 91.11% and 90.00%, respectively. Additionally, when experiments are conducted using the whole dataset, an increased accuracy of 7.31% and 4.23% for the binary classification of valence and arousal, and an improved accuracy of 5.36% for the four classifications of valence-arousal are achieved when TCMA is used in modal fusion, compared to using only facial expression modality, which fully demonstrates the effectiveness and robustness of TCMA. This method makes it possible to realize multimodal emotion recognition of facial expressions and contactless physiological signals in reality.

Talking Face Generation With Audio-Deduced Emotional Landmarks.

The goal of talking face generation is to synthesize a sequence of face images of the specified identity, ensuring the mouth movements are synchronized with the given audio. Recently, image-based talking face generation has emerged as a popular approach. It could generate talking face images synchronized with the audio merely depending on a facial image of arbitrary identity and an audio clip. Despite the accessible input, it forgoes the exploitation of the audio emotion, inducing the generated faces to suffer from emotion unsynchronization, mouth inaccuracy, and image quality deficiency. In this article, we build a bistage audio emotion-aware talking face generation (AMIGO) framework, to generate high-quality talking face videos with cross-modally synced emotion. Specifically, we propose a sequence-to-sequence (seq2seq) cross-modal emotional landmark generation network to generate vivid landmarks, whose lip and emotion are both synchronized with input audio. Meantime, we utilize a coordinated visual emotion representation to improve the extraction of the audio one. In stage two, a feature-adaptive visual translation network is designed to translate the synthesized landmarks into facial images. Concretely, we proposed a feature-adaptive transformation module to fuse the high-level representations of landmarks and images, resulting in significant improvement in image quality. We perform extensive experiments on the multi-view emotional audio-visual dataset (MEAD) and crowd-sourced emotional multimodal actors dataset (CREMA-D) benchmark datasets, demonstrating that our model outperforms state-of-the-art benchmarks.

A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision

Mild Cognitive Impairment (MCI) is an early stage of memory loss or other cognitive ability loss in individuals who maintain the ability to independently perform most activities of daily living. It is considered a transitional stage between normal cognitive stage and more severe cognitive declines like dementia or Alzheimer’s. Based on the reports from the National Institute of Aging (NIA), people with MCI are at a greater risk of developing dementia, thus it is of great importance to detect MCI at the earliest possible to mitigate the transformation of MCI to Alzheimer’s and dementia. Recent studies have harnessed Artificial Intelligence (AI) to develop automated methods to predict and detect MCI. The majority of the existing research is based on unimodal data (e.g., only speech or prosody), but recent studies have shown that multimodality leads to a more accurate prediction of MCI. However, effectively exploiting different modalities is still a big challenge due to the lack of efficient fusion methods. This study proposes a robust fusion architecture utilizing an embedding-level fusion via a co-attention mechanism to leverage multimodal data for MCI prediction. This approach addresses the limitations of early and late fusion methods, which often fail to preserve inter-modal relationships. Our embedding-level fusion aims to capture complementary information across modalities, enhancing predictive accuracy. We used the I-CONECT dataset, where a large number of semi-structured conversations via internet/webcam between participants aged 75+ years old and interviewers were recorded. We introduce a multimodal speech-language-vision Deep Learning-based method to differentiate MCI from Normal Cognition (NC). Our proposed architecture includes co-attention blocks to fuse three different modalities at the embedding level to find the potential interactions between speech (audio), language (transcribed speech), and vision (facial videos) within the cross-Transformer layer. Experimental results demonstrate that our fusion method achieves an average AUC of 85.3% in detecting MCI from NC, significantly outperforming unimodal (60.9%) and bimodal (76.3%) baseline models. This superior performance highlights the effectiveness of our model in capturing and utilizing the complementary information from multiple modalities, offering a more accurate and reliable approach for MCI prediction.

Multimodal Fusion for Talking Face Generation Utilizing Speech-Related Facial Action Units

Talking face generation is to synthesize a lip-synchronized talking face video by inputting an arbitrary face image and corresponding audio clips. The current talking face model can be divided into four parts: visual feature extraction, audio feature processing, multimodal feature fusion, and rendering module. For the visual feature extraction part, existing methods face the challenge of complex learning task with noisy features, this article introduces an attention-based disentanglement module to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. For the multimodal feature fusion part, existing methods ignore not only the interaction and relationship of cross-modal information but also the local driving information of the mouth muscles. This study proposes a novel generative framework that incorporates a dilated non-causal temporal convolutional self-attention network as a multimodal fusion module to enhance the learning of cross-modal features. The proposed method employs both audio- and speech-related facial AUs as driving information. Speech-related AU information can facilitate more accurate mouth movements. Given the high correlation between speech and speech-related AUs, we propose an audio-to-AU module to predict speech-related AU information. Finally, we present a diffusion model for the synthesis of talking face images. We verify the effectiveness of the proposed model on the GRID and TCD-TIMIT datasets. An ablation study is also conducted to verify the contribution of each component. The results of quantitative and qualitative experiments demonstrate that our method outperforms existing methods in terms of both image quality and lip-sync accuracy. Code is available at https://mftfg-au.github.io/Multimodal_Fusion/.

Mild cognitive impairment prediction based on multi-stream convolutional neural networks

BackgroundMild cognitive impairment (MCI) is the transition stage between the cognitive decline expected in normal aging and more severe cognitive decline such as dementia. The early diagnosis of MCI plays an important role in human healthcare. Current methods of MCI detection include cognitive tests to screen for executive function impairments, possibly followed by neuroimaging tests. However, these methods are expensive and time-consuming. Several studies have demonstrated that MCI and dementia can be detected by machine learning technologies from different modality data. This study proposes a multi-stream convolutional neural network (MCNN) model to predict MCI from face videos.ResultsThe total effective data are 48 facial videos from 45 participants, including 35 videos from normal cognitive participants and 13 videos from MCI participants. The videos are divided into several segments. Then, the MCNN captures the latent facial spatial features and facial dynamic features of each segment and classifies the segment as MCI or normal. Finally, the aggregation stage produces the final detection results of the input video. We evaluate 27 MCNN model combinations including three ResNet architectures, three optimizers, and three activation functions. The experimental results showed that the ResNet-50 backbone with Swish activation function and Ranger optimizer produces the best results with an F1-score of 89% at the segment level. However, the ResNet-18 backbone with Swish and Ranger achieves the F1-score of 100% at the participant level.ConclusionsThis study presents an efficient new method for predicting MCI from facial videos. Studies have shown that MCI can be detected from facial videos, and facial data can be used as a biomarker for MCI. This approach is very promising for developing accurate models for screening MCI through facial data. It demonstrates that automated, non-invasive, and inexpensive MCI screening methods are feasible and do not require highly subjective paper-and-pencil questionnaires. Evaluation of 27 model combinations also found that ResNet-50 with Swish is more stable for different optimizers. Such results provide directions for hyperparameter tuning to further improve MCI predictions.

Beyond the visible: thermal data for facial soft biometric estimation

In recent years, the estimation of biometric parameters from facial visuals, including images and videos, has emerged as a prominent area of research. However, the robustness of deep learning-based models is challenged, particularly in the presence of changing illumination conditions. To overcome these limitations and unlock new opportunities, thermal imagery has arisen as a viable alternative. Nevertheless, the limited availability of datasets containing thermal data and the small amount of annotations on them limits the exploration of this spectrum. Motivated by this gap, this paper introduces the Label-EURECOM Visible and Thermal (LVT) Face Dataset for face biometrics. This pioneering dataset includes paired visible and thermal images and videos from 52 subjects along with metadata of 22 soft biometrics and health parameters. Due to the reduced number of existing datasets in this domain, the LVT Face Dataset aims to facilitate further research and advancements in the utilization of thermal imagery for diverse eHealth applications and soft biometric estimation. Moreover, we present the first comparative study between visible and thermal spectra as input images for soft biometric estimation, namely gender age and weight, from face images on our collected dataset.

Generalized face forgery detection with self-supervised face geometry information analysis network

The emergence of high-quality deepfake facial videos has raised concerns about the security of facial images. Existing face forgery detectors mainly tend to locate a specific forgery region of the human face for detection, which achieves satisfactory performance with known forgery patterns presented in the training set. However, with the continuous advancements in face forgery technology, this approach becomes less reliable with new forgery patterns that emerge. Towards this end, we proposed a novel Self-supervised Face Geometry Information Analysis Network (SF-GAN) method for generalized face forgery detection. SF-GAN effectively leverages the relationships among informative regions based on information theory. Drawing from information theory, regions with high uncertainty tend to contain more valuable information. Our methodology integrates a self-supervised learning mechanism, enabling the precise identification of multiple informative regions. Furthermore, we leverage facial geometry by establishing both explicit and latent geometric relationships through the use of Graph Convolutional Networks (GCNs). Within our framework, facial landmarks and informative regions are depicted as nodes in the GCNs. By analyzing the geometric relationships between the graph of facial landmarks and the graph of informative regions, we are able to identify valid anomalous regions, thereby minimizing uncertainty. Our proposed model gains a comprehensive understanding of common information in face forgery images. Extensive experiments on eight large-scale benchmark datasets: FaceForensics++ (FF++), WildDeepfake (WDF), Celeb-DF v2 (CDF), DeepFake Detection Challenge (DFDC), DFDC preview (DFDC-P), Deepfake Detection (DFD), DeeperForensics-1.0 (DF-1.0) and ForgeryNIR, show that the proposed method is comparable to state-of-the-arts and exhibits better generalizability. Specifically, our SF-GAN, when trained on high-quality FF++ data, achieves an impressive AUC of 76.43% on the CDF dataset.

Facial Video-Based Non-Contact Stress Recognition Utilizing Multi-Task Learning With Peak Attention.

Negative emotional states, such as anxiety and depression, pose significant challenges in contemporary society, often stemming from the stress encountered in daily activities. Stress (state or level) recognition is a crucial prerequisite for effective stress management and intervention. Presently, wearable devices have been employed to capture physiological signals and analyze stress states. However, their constant skin contact can lead to discomfort and disturbance during prolonged monitoring. In this paper, a peak attention-based multitasking framework is presented for non-contact stress recognition. The framework extracts rPPG signals from RGB facial videos, utilizing them as inputs for a novel multi-task attentional convolutional neural network for stress recognition (MTASR). It incorporates peak detection and HR estimation as auxiliary tasks to facilitate stress recognition. By leveraging multi-task learning, MTASR can utilize information related to stress physiological responses, thereby enhancing feature extraction efficiency. For stress recognition, two binary classification tasks are applied: stress state recognition and stress level recognition. The model is validated on the UBFC-Phys public dataset and demonstrates an accuracy of 94.33% for stress state recognition and 83.83% for stress level recognition. The proposed method outperforms the dataset's baseline methods and other competing approaches.