Face Video Research Articles

HUMAN FOLLOWING CART

The development of a human-following robot utilizing Raspberry Pi and camera technology represents a significant advancement in autonomous robotics, offering practical solutions for various industries. This project integrates a Raspberry Pi microcomputer, a camera, and ultrasonic sensors to create a robot capable of tracking and following a human operator. The system employs computer vision algorithms, facilitated by a Haar Cascade Classifier, to detect faces in real-time video feeds. The video feed is processed to determine the position of the human within the frame, enabling the robot to navigate accordingly. Ultrasonic sensors are used to measure the distance between the robot and the human, ensuring that a safe and optimal following distance is maintained. The motors and motor drivers are controlled based on the visual and distance data, allowing the robot to move forward, backward, or turn as needed. This combination of technologies enhances worker safety by reducing physical strain and improves operational efficiency by automating the transportation of goods, tools, and supplies. The implementation of this system demonstrates the practical application of affordable and accessible technologies like the Raspberry Pi in creating intelligent, autonomous robots. The project highlights the potential for such robots to revolutionize workflows in warehouses, manufacturing plants, and hospitals, making advanced automation more attainable for a wide range of businesses. The integration of computer vision and sensor-based navigation in human-following robots marks a significant step forward in the evolution of intelligent workspaces and autonomous systems.

Deepfake Video Detection Using Facial Feature Points and Ch-Transformer

With the development of Metaverse technology, the avatar in Metaverse has faced serious security and privacy concerns. Analyzing facial features to distinguish between genuine and manipulated facial videos holds significant research importance for ensuring the authenticity of characters in the virtual world and for mitigating discrimination, as well as preventing malicious use of facial data. To address this issue, the Facial Feature Points and Ch-Transformer (FFP-ChT) deepfake video detection model is designed based on the clues of different facial feature points distribution in real and fake videos and different displacement distances of real and fake facial feature points between frames. The face video input is first detected by the BlazeFace model, and the face detection results are fed into the FaceMesh model to extract 468 facial feature points. Then the Lucas-Kanade (LK) optical flow method is used to track the points of the face, the face calibration algorithm is introduced to re-calibrate the facial feature points, and the jitter displacement is calculated by tracking the facial feature points between frames. Finally, the Class-head (Ch) is designed in the transformer, and the facial feature points and facial feature point displacement are jointly classified through the Ch-Transformer model. In this way, the designed Ch-Transformer classifier is able to accurately and effectively identify deepfake videos. Experiments on open datasets clearly demonstrate the effectiveness and generalization capabilities of our approach.

DCM2Net: an improved face recognition model for panoramic stereoscopic videos.

The panoramic stereo video has brought a new visual experience for the audience with its immersion and stereo effect. In panoramic stereo video, the face is an important element. However, the face image in panoramic stereo video has varying degrees of deformation. This brings new challenges to face recognition. Therefore, this paper proposes a face recognition model DCM2Net (Deformable Convolution MobileFaceNet) for panoramic stereo video. The model mainly integrates the feature information between channels during feature fusion, redistributes the information between channels in the deeper part of the network, and fully uses the information between different channels for feature extraction. This paper also built a panoramic stereo video live system, using the DCM2Net model to recognize the face in panoramic stereo video, and the recognition results are displayed in the video. After experiments on different datasets, the results show that our model has better results on popular datasets and panoramic datasets.

Generating dynamic lip-syncing using target audio in a multimedia environment

The presented research focuses on the challenging task of creating lip-sync facial videos that align with a specified target speech segment. A novel deep-learning model has been developed to produce precise synthetic lip movements corresponding to the speech extracted from an audio source. Consequently, there are instances where portions of the visual data may fall out of sync with the updated audio and this challenge is handled through, a novel strategy, leveraging insights from a robust lip-sync discriminator. Additionally, this study introduces fresh criteria and evaluation benchmarks for assessing lip synchronization in unconstrained videos. LipChanger demonstrates improved PSNR values, indicative of enhanced image quality. Furthermore, it exhibits highly accurate lip synthesis, as evidenced by lower LMD values and higher SSIM values. These outcomes suggest that the LipChanger approach holds significant potential for enhancing lip synchronization in talking face videos, resulting in more realistic lip movements. The proposed LipChanger model and its associated evaluation benchmarks show promise and could potentially contribute to advancements in lip-sync technology for unconstrained talking face videos.

Channel-Wise Interactive Learning for Remote Heart Rate Estimation From Facial Video

Channel-Wise Interactive Learning for Remote Heart Rate Estimation From Facial Video

Leveraging 3D convolutional neural network and 3D visible-near-infrared multimodal imaging for enhanced contactless oximetry.

Monitoring oxygen saturation ( ) is important in healthcare, especially for diagnosing and managing pulmonary diseases. Non-contact approaches broaden the potential applications of measurement by better hygiene, comfort, and capability for long-term monitoring. However, existing studies often encounter challenges such as lower signal-to-noise ratios and stringent environmental conditions. We aim to develop and validate a contactless measurement approach using 3D convolutional neural networks (3D CNN) and 3D visible-near-infrared (VIS-NIR) multimodal imaging, to offer a convenient, accurate, and robust alternative for monitoring. We propose an approach that utilizes a 3D VIS-NIR multimodal camera system to capture facial videos, in which is estimated through 3D CNN by simultaneously extracting spatial and temporal features. Our approach includes registration of multimodal images, tracking of the 3D region of interest, spatial and temporal preprocessing, and 3D CNN-based feature extraction and regression. In a breath-holding experiment involving 23 healthy participants, we obtained multimodal video data with reference values ranging from 80% to 99% measured by pulse oximeter on the fingertip. The approach achieved a mean absolute error (MAE) of 2.31% and a Pearson correlation coefficient of 0.64 in the experiment, demonstrating good agreement with traditional pulse oximetry. The discrepancy of estimated values was within 3% of the reference for of all 1-s time points. Besides, in clinical trials involving patients with sleep apnea syndrome, our approach demonstrated robust performance, with an MAE of less than 2% in estimations compared to gold-standard polysomnography. The proposed approach offers a promising alternative for non-contact oxygen saturation measurement with good sensitivity to desaturation, showing potential for applications in clinical settings.

A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online

Recent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation is a custom extension written for jsPsych, an existing JavaScript framework for running behavioral experiments online. This extension gathers data from the participant’s webcam and processes them through FaceMesh, an open-source, JavaScript face-tracker. Face-tracking happens on the fly inside the participant’s browser. Only time-stamped vertical or horizontal lip apertures are saved to the experiment server. That is, this extension allows experiments implemented in jsPsych to collect de-identified lip kinematics from participants seated at their own home computers. After explaining the core functionality of the extension, this paper presents two validation experiments. The first establishes that utterances with different lip kinematics result in very different lip aperture trajectories, and that the timing of a key kinematic landmark agrees with the timing of acoustic landmarks obtained for the same utterances. The second experiment replicates a vowel-priming result previously demonstrated using a prior face-tracking system and saved facial video. All associated scripts have been made publicly available.

Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language.

When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers' visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers' latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

Automatic detection of facial expressions during the Cyberball paradigm in Borderline Personality Disorder: a pilot study.

Borderline Personality Disorder (BPD) symptoms include inappropriate control of anger and severe emotional dysregulation after rejection in daily life. Nevertheless, when using the Cyberball paradigm, a tossing game to simulate social exclusion, the seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and contempt) have not been exhaustively tracked out. It was hypothesized that these patients would show anger, contempt, and disgust during the condition of exclusion versus the condition of inclusion. When facial emotions are automatically detected by Artificial Intelligence, "blending", -or a mixture of at least two emotions- and "masking", -or showing happiness while expressing negative emotions- may be most easily traced expecting higher percentages during exclusion rather than inclusion. Therefore, face videos of fourteen patients diagnosed with BPD (26 ± 6 years old), recorded while playing the tossing game, were analyzed by the FaceReader software. The comparison of conditions highlighted an interaction for anger: it increased during inclusion and decreased during exclusion. During exclusion, the masking of surprise; i.e., displaying happiness while feeling surprised, was significantly more expressed. Furthermore, disgust and contempt were inversely correlated with greater difficulties in emotion regulation and symptomatology, respectively. Therefore, the automatic detection of emotional expressions during both conditions could be useful in rendering diagnostic guidelines in clinical scenarios.

A Study on the Screening of Children at Risk for Developmental Disabilities Using Facial Landmarks Derived From a Mobile-Based Application

Objective Early detection and intervention of developmental disabilities (DDs) are critical to improving the long-term outcomes of afflicted children. In this study, our objective was to utilize facial landmark features from mobile application to distinguish between children with DDs and typically developing (TD) children.Methods The present study recruited 89 children, including 33 diagnosed with DD, and 56 TD children. The aim was to examine the effectiveness of a deep learning classification model using facial video collected from children through mobile-based application. The study participants underwent comprehensive developmental assessments, which included the child completion of the Korean Psychoeducational Profile-Revised and caregiver completing the Korean versions of Vineland Adaptive Behavior Scale, Korean version of the Childhood Autism Rating Scale, Social Responsiveness Scale, and Child Behavior Checklist. We extracted facial landmarks from recorded videos using mobile application and performed DDs classification using long short-term memory with stratified 5-fold cross-validation.Results The classification model shows an average accuracy of 0.88 (range: 0.78–1.00), an average precision of 0.91 (range: 0.75–1.00), and an average F1-score of 0.80 (range: 0.60–1.00). Upon interpreting prediction results using SHapley Additive exPlanations (SHAP), we verified that the most crucial variable was the nodding head angle variable, with a median SHAP score of 2.6. All the top 10 contributing variables exhibited significant differences in distribution between children with DD and TD (p<0.05).Conclusion The results of this study provide evidence that facial landmarks, utilizing readily available mobile-based video data, can be used to detect DD at an early stage.

PointTransform Networks for automatic depression level prediction via facial keypoints

According to physiological reports, individuals with different levels of depression present various facial dynamic patterns. Thus, researchers have examined facial changes to predict depression severity. However, they often use facial images or videos of the subjects, which increases the risk of privacy leakage. Therefore, we intend to use the mixer layer to process facial keypoint sequences to predict the depression levels. However, the mixer layer cannot guarantee that the output and input sequences have the same temporal properties, which prevents the residual connection from being additive in a physical sense. To this end, we construct a PointTransform Network (PTN). In the constructed model, a Mixer Attention Layer (MAL) and a Token Sequence Aggregation (TSA) module are proposed. The proposed MAL embeds the results of channel-communication and token-communication into the sequence in the form of weights through the attention mechanism, which maintains the temporal order among tokens in the sequence during the calculation process and compensates for the limitations of the mixer layer. The proposed TSA module can integrate the depression level prediction results of all facial keypoints through the attention mechanism. Therefore, the TSA module achieves the unity of decision-level fusion and tensor vectorization. Experiments are conducted on several benchmark databases and the results demonstrate the effectiveness of our method.

Consistent movement of viewers' facial keypoints while watching emotionally evocative videos.

Neuropsychological research aims to unravel how diverse individuals' brains exhibit similar functionality when exposed to the same stimuli. The evocation of consistent responses when different subjects watch the same emotionally evocative stimulus has been observed through modalities like fMRI, EEG, physiological signals and facial expressions. We refer to the quantification of these shared consistent signals across subjects at each time instant across the temporal dimension as Consistent Response Measurement (CRM). CRM is widely explored through fMRI, occasionally with EEG, physiological signals and facial expressions using metrics like Inter-Subject Correlation (ISC). However, fMRI tools are expensive and constrained, while EEG and physiological signals are prone to facial artifacts and environmental conditions (such as temperature, humidity, and health condition of subjects). In this research, facial expression videos are used as a cost-effective and flexible alternative for CRM, minimally affected by external conditions. By employing computer vision-based automated facial keypoint tracking, a new metric similar to ISC, called the Average t-statistic, is introduced. Unlike existing facial expression-based methodologies that measure CRM of secondary indicators like inferred emotions, keypoint, and ICA-based features, the Average t-statistic is closely associated with the direct measurement of consistent facial muscle movement using the Facial Action Coding System (FACS). This is evidenced in DISFA dataset where the time-series of Average t-statistic has a high correlation (R2 = 0.78) with a metric called AU consistency, which directly measures facial muscle movement through FACS coding of video frames. The simplicity of recording facial expressions with the automated Average t-statistic expands the applications of CRM such as measuring engagement in online learning, customer interactions, etc., and diagnosing outliers in healthcare conditions like stroke, autism, depression, etc. To promote further research, we have made the code repository publicly available.

Mild cognitive impairment detection from facial video interviews by applying spatial-to-temporal attention module

Early detection of Mild Cognitive Impairment (MCI) leads to early interventions to slow the progression from MCI into dementia. Deep Learning (DL) algorithms could help achieve early non-invasive and low-cost detection of MCI. This paper presents the detection of MCI in older adults using DL models based only on facial features extracted from video-recorded conversations at home. We used the data collected from the I-CONECT behavioral intervention study (NCT02871921), where several sessions of semi-structured interviews between socially isolated older individuals and interviewers were video recorded. We develop a framework that extracts holistic spatial facial features using a convolutional autoencoder and temporal information using transformers. We proposed the Spatial-to-Temporal Attention Module (STAM) to detect the I-CONECT study participants’ cognitive conditions (MCI vs. those with normal cognition (NC)) using facial and interaction features. The interaction features of the facial features improved the prediction performance compared with applying facial features solely. The detection accuracy using this combined method reached 88%, whereas the accuracy without applying the segments and sequences information of the facial features within a video on a certain theme was 84%. Overall, the results show that spatiotemporal facial features modeled using DL algorithms have a discriminating power for the detection of MCI.

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition

AbstractContinuous emotion recognition is to predict emotion states through affective information and more focus on the continuous variation of emotion. Fusion of electroencephalography (EEG) and facial expressions videos has been used in this field, while there are with some limitations in current researches, such as hand‐engineered features, simple approaches to integration. Hence, a new continuous emotion recognition model is proposed based on the fusion of EEG and facial expressions videos named residual multimodal Transformer (RMMT). Firstly, the Resnet50 and temporal convolutional network (TCN) are utilised to extract spatiotemporal features from videos, and the TCN is also applied to process the computed EEG frequency power to acquire spatiotemporal features of EEG. Then, a multimodal Transformer is used to fuse the spatiotemporal features from the two modalities. Furthermore, a residual connection is introduced to fuse shallow features with deep features which is verified to be effective for continuous emotion recognition through experiments. Inspired by knowledge distillation, the authors incorporate feature‐level loss into the loss function to further enhance the network performance. Experimental results show that the RMMT reaches a superior performance over other methods for the MAHNOB‐HCI dataset. Ablation studies on the residual connection and loss function in the RMMT demonstrate that both of them is functional.

Maternal childhood emotional abuse increases cardiovascular responses to children's emotional facial expressions.

Parents with a history of childhood maltreatment may be more likely to respond inadequately to their child's emotional cues, such as crying or screaming, due to previous exposure to prolonged stress. While studies have investigated parents' physiological reactions to their children's vocal expressions of emotions, less attention has been given to their responses when perceiving children's facial expressions of emotions. The present study aimed to determine if viewing facial expressions of emotions in children induces cardiovascular changes in mothers (hypo- or hyper-arousal) and whether these differ as a function of childhood maltreatment. A total of 104 mothers took part in this study. Their experiences of childhood maltreatment were measured using the Childhood Trauma Questionnaire (CTQ). Participants' electrocardiogram signals were recorded during a task in which they viewed a landscape video (baseline) and images of children's faces expressing different intensities of emotion. Heart rate variability (HRV) was extracted from the recordings as an indicator of parasympathetic reactivity. Participants presented two profiles: one group of mothers had a decreased HRV when presented with images of children's facial expressions of emotions, while the other group's HRV increased. However, HRV change was not significantly different between the two groups. The interaction between HRV groups and the severity of maltreatment experienced was marginal. Results suggested that experiences of childhood emotional abuse were more common in mothers whose HRV increased during the task. Therefore, more severe childhood experiences of emotional abuse could be associated with mothers' cardiovascular hyperreactivity. Maladaptive cardiovascular responses could have a ripple effect, influencing how mothers react to their children's facial expressions of emotions. That reaction could affect the quality of their interaction with their child. Providing interventions that help parents regulate their physiological and behavioral responses to stress might be helpful, especially if they have experienced childhood maltreatment.

Changes in Perceived Emotions in Facial Paralysis Patients After Depressor Anguli Oris Excision.

Depressor anguli oris (DAO) excision can improve clinician-graded, objective, and patient-reported smile outcomes in patients with nonflaccid facial paralysis (NFFP). However, no prior research has studied changes in perceived emotions after surgery. This study quantifies changes in perceived emotions with smiling after DAO excision in the largest case series presented to date. Prospectively collected data from patients with NFFP who underwent DAO excision at a tertiary care facial nerve center were reviewed. Patient-reported, clinician-graded, and objective smile metrics were compared before and after surgery. Videos of faces at rest and while smiling were analyzed by artificial intelligence-derived facial expression analysis software to quantify perceived emotions. Sixty-eight patients underwent isolated DAO excision between August 2021 and August 2023. Patients conveyed significantly more perceived happiness with smile and at rest after surgery (p < 0.001 and p = 0.012, respectively). DAO excision improved oral commissure excursion (p < 0.001), dental show (p < 0.001), and smile angle (p < 0.001) symmetry. Patients reported significant improvements in smiling and social function after surgery. This study demonstrates DAO excision increases perceived happiness conveyed by patients with NFFP while smiling and at rest. It confirms improved objective, clinician-graded, and patient-reported smile outcomes after surgery. 4 Laryngoscope, 134:4028-4035, 2024.

A novel temporal adaptive fuzzy neural network for facial feature based fatigue assessment

When engaging in activities such as using video display terminals, driving, or sports, effective fatigue monitoring is crucial. Nevertheless, obtaining biological information through contact devices may interfere with normal activities. Deterministic expressions in mainstream machine learning have limitations in reflecting the continuous, dynamic, and fuzzy process of fatigue status. Additionally, fatigue is a cumulative process where the previous state impacts the assessment of the current one. To address these needs and challenges, this paper proposes a novel model based on facial videos called the temporal adaptive fuzzy neural network (TAFNN) for fatigue assessment. TAFNN utilizes an adaptive fuzzy neural network as its foundation and employs causal and dilated convolutions for the time information processing method to achieve time series extraction and fatigue assessment. It leverages facial physiological and motion features to minimize interference during assessment. Furthermore, TAFNN introduces a new calculation method for rule antecedents to enhance stability. Experimental results demonstrate that TAFNN effectively captures the cumulation and fuzziness of state changes, outperforming other widely adopted methods in both assessment ability and runtime performance. The improved rule antecedent calculation method successfully mitigates the issue of multiple memberships rapidly approaching zero after combination. Through 1000 repeated experiments, the enhanced method reduces TAFNN’s instability by 81.58%.

HRUNet: Assessing Uncertainty in Heart Rates Measured From Facial Videos.

Video-based Photoplethysmography (VPPG) offers the capability to measure heart rate (HR) from facial videos. However, the reliability of the HR values extracted through this method remains uncertain, especially when videos are affected by various disturbances. Confronted by this challenge, we introduce an innovative framework for VPPG-based HR measurements, with a focus on capturing diverse sources of uncertainty in the predicted HR values. In this context, a neural network named HRUNet is structured for HR extraction from input facial videos. Departing from the conventional training approach of learning specific weight (and bias) values, we leverage the Bayesian posterior estimation to derive weight distributions within HRUNet. These distributions allow for sampling to encode uncertainty stemming from HRUNet's limited performance. On this basis, we redefine HRUNet's output as a distribution of potential HR values, as opposed to the traditional emphasis on the single most probable HR value. The underlying goal is to discover the uncertainty arising from inherent noise in the input video. HRUNet is evaluated across 1,098 videos from seven datasets, spanning three scenarios: undisturbed, motion-disturbed, and light-disturbed. The ensuing test outcomes demonstrate that uncertainty in the HR measurements increases significantly in the scenarios marked by disturbances, compared to that in the undisturbed scenario. Moreover, HRUNet outperforms state-of-the-art methods in HR accuracy when excluding HR values with 0.4 uncertainty. This underscores that uncertainty emerges as an informative indicator of potentially erroneous HR measurements. With enhanced reliability affirmed, the VPPG technique holds the promise for applications in safety-critical domains.

DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation.

Predominant techniques on talking head generation largely depend on 2D information, including facial appearances and motions from input face images. Nevertheless, dense 3D facial geometry, such as pixel-wise depth, plays a critical role in constructing accurate 3D facial structures and suppressing complex background noises for generation. However, dense 3D annotations for facial videos is prohibitively costly to obtain. In this paper, first, we present a novel self-supervised method for learning dense 3D facial geometry (i.e., depth) from face videos, without requiring camera parameters and 3D geometry annotations in training. We further propose a strategy to learn pixel-level uncertainties to perceive more reliable rigid-motion pixels for geometry learning. Second, we design an effective geometry-guided facial keypoint estimation module, providing accurate keypoints for generating motion fields. Lastly, we develop a 3D-aware cross-modal (i.e., appearance and depth) attention mechanism, which can be applied to each generation layer, to capture facial geometries in a coarse-to-fine manner. Extensive experiments are conducted on three challenging benchmarks (i.e., VoxCeleb1, VoxCeleb2, and HDTF). The results demonstrate that our proposed framework can generate highly realistic-looking reenacted talking videos, with new state-of-the-art performances established on these benchmarks.

Refining Localized Attention Features with Multi-Scale Relationships for Enhanced Deepfake Detection in Spatial-Frequency Domain

The rapid advancement of deep learning and large-scale AI models has simplified the creation and manipulation of deepfake technologies, which generate, edit, and replace faces in images and videos. This gradual ease of use has turned the malicious application of forged faces into a significant threat, complicating the task of deepfake detection. Despite the notable success of current deepfake detection methods, which predominantly employ data-driven CNN classification models, these methods exhibit limited generalization capabilities and insufficient robustness against novel data unseen during training. To tackle these challenges, this paper introduces a novel detection framework, ReLAF-Net. This framework employs a restricted self-attention mechanism that applies self-attention to deep CNN features flexibly, facilitating the learning of local relationships and inter-regional dependencies at both fine-grained and global levels. This attention mechanism has a modular design that can be seamlessly integrated into CNN networks to improve overall detection performance. Additionally, we propose an adaptive local frequency feature extraction algorithm that decomposes RGB images into fine-grained frequency domains in a data-driven manner, effectively isolating fake indicators in the frequency space. Moreover, an attention-based channel fusion strategy is developed to amalgamate RGB and frequency information, achieving a comprehensive facial representation. Tested on the high-quality version of the FaceForensics++ dataset, our method attained a detection accuracy of 97.92%, outperforming other approaches. Cross-dataset validation on Celeb-DF, DFDC, and DFD confirms the robust generalizability, offering a new solution for detecting high-quality deepfake videos.