Videos Of Facial Expressions Research Articles

DEEP Q-NETWORK (DQN) PARTIALOCCLUSION SEGMENTATION AND BACKTRACKING SEARCH OPTIMIZATION ALGORITHM (BSOA) WITH OPTICAL FLOW RECONSTRUCTION FOR FACIAL EXPRESSION EMOTION RECOGNITION

Video facial expression recognition (FER) has garnered a lot of attention recently and is helpful for several applications. Although many algorithms demonstrate impressive performance in a controlled environment without occlusion, identification in the presence of partial facial occlusion remains a challenging issue. Solutions based on reconstructing the obscured area of the face have been suggested as a way to deal with occlusions. These options mostly rely on the face’s shape or texture. Nonetheless, the resemblance in facial expressions among individuals appears to be a valuable advantage for the reconstruction. For semantic segmentation based on occlusions, Reinforcement Learning (RL) is introduced as the initial stage. From a pool of unlabeled data, an agent learns a policy to choose a subset of tiny informative image patches to be tagged instead of full images. In the second stage, a trained Backtracking Search Algorithm (BSA) is used to rebuild optical flows that have been distorted by the occlusion. On obtaining optical flows estimated from occluded facial frames, AEs restore optical flows of occluded regions. These recovered optical flows become inputs to anticipate classes f expressions. Optical flux reconstructions then classify stages. This study evaluates classification model’s performances for face expression identification based on Very Deep Convolution Networks (VGGNet). Furthermore, it produces more accurate confusion matrices and proposes approaches for the KMU-FED and CK+ databases, respectively. The results are evaluated using metrics including recall, f-measure, accuracy, and precision.

Sensing emotional valence and arousal dynamics through automated facial action unit analysis

Information about the concordance between dynamic emotional experiences and objective signals is practically useful. Previous studies have shown that valence dynamics can be estimated by recording electrical activity from the muscles in the brows and cheeks. However, whether facial actions based on video data and analyzed without electrodes can be used for sensing emotion dynamics remains unknown. We investigated this issue by recording video of participants’ faces and obtaining dynamic valence and arousal ratings while they observed emotional films. Action units (AUs) 04 (i.e., brow lowering) and 12 (i.e., lip-corner pulling), detected through an automated analysis of the video data, were negatively and positively correlated with dynamic ratings of subjective valence, respectively. Several other AUs were also correlated with dynamic valence or arousal ratings. Random forest regression modeling, interpreted using the SHapley Additive exPlanation tool, revealed non-linear associations between the AUs and dynamic ratings of valence or arousal. These results suggest that an automated analysis of facial expression video data can be used to estimate dynamic emotional states, which could be applied in various fields including mental health diagnosis, security monitoring, and education.

Consistent movement of viewers' facial keypoints while watching emotionally evocative videos.

Neuropsychological research aims to unravel how diverse individuals' brains exhibit similar functionality when exposed to the same stimuli. The evocation of consistent responses when different subjects watch the same emotionally evocative stimulus has been observed through modalities like fMRI, EEG, physiological signals and facial expressions. We refer to the quantification of these shared consistent signals across subjects at each time instant across the temporal dimension as Consistent Response Measurement (CRM). CRM is widely explored through fMRI, occasionally with EEG, physiological signals and facial expressions using metrics like Inter-Subject Correlation (ISC). However, fMRI tools are expensive and constrained, while EEG and physiological signals are prone to facial artifacts and environmental conditions (such as temperature, humidity, and health condition of subjects). In this research, facial expression videos are used as a cost-effective and flexible alternative for CRM, minimally affected by external conditions. By employing computer vision-based automated facial keypoint tracking, a new metric similar to ISC, called the Average t-statistic, is introduced. Unlike existing facial expression-based methodologies that measure CRM of secondary indicators like inferred emotions, keypoint, and ICA-based features, the Average t-statistic is closely associated with the direct measurement of consistent facial muscle movement using the Facial Action Coding System (FACS). This is evidenced in DISFA dataset where the time-series of Average t-statistic has a high correlation (R2 = 0.78) with a metric called AU consistency, which directly measures facial muscle movement through FACS coding of video frames. The simplicity of recording facial expressions with the automated Average t-statistic expands the applications of CRM such as measuring engagement in online learning, customer interactions, etc., and diagnosing outliers in healthcare conditions like stroke, autism, depression, etc. To promote further research, we have made the code repository publicly available.

Investigating emotion fluctuations in driving behaviors of online car-hailing drivers using naturalistic driving data

Research has shown that a driver’s emotional state is closely related to the driver’s driving behavior. However, most studies have focused on the impact of static and discrete emotions on driver behavior, neglecting the emotion fluctuations that can occur in real-world contexts. To address this gap, we analyzed emotion change patterns of drivers engaged in aggressive driving versus normal driving. We used online car-hailing data to identify aggressive driving, which is defined by a dynamic acceleration threshold determined from vehicle kinematics data. We used two dimensions, valence and arousal, to describe different levels of emotions by processing videos of drivers' facial expressions using FaceReader software. By analyzing the characteristics of drivers’ emotional changes, we determined a six-second time window as the length of an emotion fluctuation segment. Then, we applied dynamic time warping k-means clustering as a time-series clustering method to divide the emotion fluctuations into several categories. We categorized the valence fluctuations into negative, calm, and positive and the arousal fluctuations into high and low. The clustering analysis revealed a marked fluctuation in the emotional valence of drivers during aggressive driving as opposed to the relatively smooth patterns observed during normal driving. Moreover, during aggressive driving, the driver’s arousal can continuously reach higher values compared to during normal driving. This study presents a novel approach to investigating the relationship between emotions and driving behavior and provides important reference and guidance for drivers’ emotional perceptions and the detection of emotion fluctuations for advanced assisted driving systems to improve driving safety.

Depression Level Assessment based on 3D CNN and Facial Expression Videos

Depression Level Assessment based on 3D CNN and Facial Expression Videos

Multi-geometry embedded transformer for facial expression recognition in videos

Dynamic facial expressions in videos express more realistic emotional states, and recognizing emotions from in-the-wild facial expression videos is a challenging task due to the changeable posture, partial occlusion and various light conditions. Although current methods have designed transformer-based models to learn spatial–temporal features, they cannot explore useful local geometry structures from both spatial and temporal views to capture subtle emotional features for the videos with varied poses and facial occlusion. To this end, we propose a novel multi-geometry embedded transformer (MGET), which adapts multi-geometry knowledge into transformers and excavates spatial–temporal geometry information as complementary to learn effective emotional features. Specifically, from a new perspective, we first design a multi-geometry distance learning (MGDL) to capture emotion-related geometry structure knowledge under Euclidean and Hyperbolic spaces. Especially based on the advantages of hyperbolic geometry, it finds the more subtle emotional changes among local spatial and temporal features. Secondly, we combine MGDL with transformer to design spatial–temporal MGETs, which capture important spatial and temporal multi-geometry features to embed them into their corresponding original features, and then perform cross-regions and cross-frame interaction on these multi-level features. Finally, MGET gains superior performance on DFEW, FERV39k and AFEW datasets, where the unweighted average recall (UAR) and weighted average recall (WAR) are 58.65%/69.91%, 41.91%/50.76% and 53.23%/55.40%, respectively, and the gained improvements are 2.55%/0.66%, 3.69%/2.63% and 3.66%/1.14% compared to M3DFEL, Logo-Forme and EST methods.

Quantifying Environmental Sensitivity Using Morphing Facial Expression Videos

Quantifying Environmental Sensitivity Using Morphing Facial Expression Videos

A multi-modal driver emotion dataset and study: Including facial expressions and synchronized physiological signals

To address the limitations of databases in the field of emotion recognition and to cater to the trend of integrating data from multiple sources, we have established a multi-modal emotional dataset based on spontaneous expression of drivers. By selecting emotional induction materials and inducing emotions before each driving task, facial expression videos and synchronous physiological signals of the drivers during driving were collected. The dataset includes records of 64 participants under five different emotions (neutral, happy, angry, sad, and fear), and the emotional valence, arousal, and peak time of all participants in each driving task were recorded. To analyze the dataset, spatio-temporal convolutional neural networks were designed to analyze the different modalities of data with varying durations in the dataset, aiming to investigate their performance in emotion recognition. The results demonstrate that the fusion of multi-modal data significantly improves the accuracy of driver's emotion recognition, with accuracy increases of 11.28% and 6.83% compared to using only facial video signals or physiological signals, respectively. Therefore, the publication and analysis of multi-modal emotional data for driving scenarios is crucial to support further research in the fields of multimodal perception and intelligent transportation engineering.

Semantic speech analysis using machine learning and deep learning techniques: a comprehensive review

Human cognitive functions such as perception, attention, learning, memory, reasoning, and problem-solving are all significantly influenced by emotion. Emotion has a particularly potent impact on attention, modifying its selectivity in particular and influencing behavior and action motivation. Artificial Emotional Intelligence (AEI) technologies enable computers to understand a user's emotional state and respond appropriately. These systems enable a realistic dialogue between people and machines. The current generation of adaptive user interference technologies is built on techniques from data analytics and machine learning (ML), namely deep learning (DL) artificial neural networks (ANN) from multimodal data, such as videos of facial expressions, stance, and gesture, voice, and bio-physiological data (such as eye movement, ECG, respiration, EEG, FMRT, EMG, eye tracking). In this study, we reviewed existing literature based on ML and data analytics techniques being used to detect emotions in speech. The efficacy of data analytics and ML techniques in this unique area of multimodal data processing and extracting emotions from speech. This study analyzes how emotional chatbots, facial expressions, images, and social media texts can be effective in detecting emotions. PRISMA methodology is used to review the existing survey. Support Vector Machines (SVM), Naïve Bayes (NB), Random Forests (RF), Recurrent Neural Networks (RNN), Logistic Regression (LR), etc., are commonly used ML techniques for emotion extraction purposes. This study provides a new taxonomy about the application of ML in SER. The result shows that Long-Short Term Memory (LSTM) and Convolutional Neural Networks (CNN) are found to be the most useful methodology for this purpose.

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding.

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark.

Extraction and analysis of factors influencing college students’ mental health based on deep learning model

Abstract This paper analyzes the factors that affect the mental health of college students, and the focus of this analysis is on emotional-emotional factors. The extroverted presentation of emotional affects is used as visual information to study the mental health status of college students. Based on the advantages of long and short memory neural networks based on deep learning models in processing two-dimensional images, a computer vision task is used to perform visual recognition, target detection, and expression image classification of college students’ facial expressions. The use of video facial expression recognition with multi-feature fusion is utilized to effectively identify the facial expression machine of college students in both laboratory-controlled and outdoor environments. The mental health status of college students was analyzed in terms of facial expression recognition and feature extraction. The recognition rate for general features was 80.3%, 89.3% for six specific facial emotions, and 84.4% for LBP features.

How can cry acoustics associate newborns' distress levels with neurophysiological and behavioral signals?

Even though infant crying is a common phenomenon in humans' early life, it is still a challenge for researchers to properly understand it as a reflection of complex neurophysiological functions. Our study aims to determine the association between neonatal cry acoustics with neurophysiological signals and behavioral features according to different cry distress levels of newborns. Multimodal data from 25 healthy term newborns were collected simultaneously recording infant cry vocalizations, electroencephalography (EEG), near-infrared spectroscopy (NIRS) and videos of facial expressions and body movements. Statistical analysis was conducted on this dataset to identify correlations among variables during three different infant conditions (i.e., resting, cry, and distress). A Deep Learning (DL) algorithm was used to objectively and automatically evaluate the level of cry distress in infants. We found correlations between most of the features extracted from the signals depending on the infant's arousal state, among them: fundamental frequency (F0), brain activity (delta, theta, and alpha frequency bands), cerebral and body oxygenation, heart rate, facial tension, and body rigidity. Additionally, these associations reinforce that what is occurring at an acoustic level can be characterized by behavioral and neurophysiological patterns. Finally, the DL audio model developed was able to classify the different levels of distress achieving 93% accuracy. Our findings strengthen the potential of crying as a biomarker evidencing the physical, emotional and health status of the infant becoming a crucial tool for caregivers and clinicians.

Human and machine recognition of dynamic and static facial expressions: prototypicality, ambiguity, and complexity.

A growing body of research suggests that movement aids facial expression recognition. However, less is known about the conditions under which the dynamic advantage occurs. The aim of this research was to test emotion recognition in static and dynamic facial expressions, thereby exploring the role of three featural parameters (prototypicality, ambiguity, and complexity) in human and machine analysis. In two studies, facial expression videos and corresponding images depicting the peak of the target and non-target emotion were presented to human observers and the machine classifier (FACET). Results revealed higher recognition rates for dynamic stimuli compared to non-target images. Such benefit disappeared in the context of target-emotion images which were similarly well (or even better) recognised than videos, and more prototypical, less ambiguous, and more complex in appearance than non-target images. While prototypicality and ambiguity exerted more predictive power in machine performance, complexity was more indicative of human emotion recognition. Interestingly, recognition performance by the machine was found to be superior to humans for both target and non-target images. Together, the findings point towards a compensatory role of dynamic information, particularly when static-based stimuli lack relevant features of the target emotion. Implications for research using automatic facial expression analysis (AFEA) are discussed.

Facial mimicry and metacognitive judgments in emotion recognition are distinctly modulated by social anxiety and autistic traits

Facial mimicry as well as the accurate assessment of one's performance when judging others’ emotional expressions have been suggested to inform successful emotion recognition. Differences in the integration of these two information sources might explain alterations in the perception of others’ emotions in individuals with Social Anxiety Disorder and individuals on the autism spectrum. Using a non-clinical sample (N = 57), we examined the role of social anxiety and autistic traits in the link between facial mimicry, or confidence in one’s performance, and emotion recognition. While participants were presented with videos of spontaneous emotional facial expressions, we measured their facial muscle activity, asked them to label the expressions and indicate their confidence in accurately labelling the expressions. Our results showed that confidence in emotion recognition was lower with higher social anxiety traits even though actual recognition was not related to social anxiety traits. Higher autistic traits, in contrast, were associated with worse recognition, and a weakened link between facial mimicry and performance. Consequently, high social anxiety traits might not affect emotion recognition itself, but the top-down evaluation of own abilities in emotion recognition contexts. High autistic traits, in contrast, may be related to lower integration of sensorimotor simulations, which promote emotion recognition.

Deep emotion change detection via facial expression analysis

Facial expressions are one of the most essential channels to communicate a person’s emotional state. In social interaction, the capability to accurately read subtle changes in facial expressions, which reveal emotional fluctuations, is critical for 1) comprehending others’ emotions in context and background situations, 2) identifying responsiveness to others’ emotions, and 3) developing social skills in human–computer interaction. In this paper, we first introduce automatic emotion change detection via facial expression that discovers timings or temporal locations in a video where facial expression significantly changes. We propose a weakly-supervised deep emotion change detection framework that does not require facial expression videos with expensive temporal annotations and instead learns static images for training. Incorporating these ideas, we performed extensive experiments to demonstrate fundamental insights into emotion change detection and the efficacy of our framework using three video datasets, i.e., CASME II, MMI, and our YoutubeECD. Furthermore, we modified our framework for temporal spotting, which is the most similar task to emotion change detection, and showed comparable results with state-of-the-art methods on CAS(ME)2, proving justification for the problem. Even though we only employed the AffectNet to train our framework rather than the CASME II, MMI, YoutubeECD, and CAS(ME)2, experimental results demonstrate its exceptional generalization capability in cross-dataset environments.

Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning models

Emotion estimation from face expression analysis is nowadays a widely-explored computer vision task. In turn, the classification of expressions relies on relevant facial features and their dynamics. Despite the promising accuracy results achieved in controlled and favorable conditions, the processing of faces acquired at a distance, entailing low-quality images, still suffers from a significant performance decrease. In particular, most approaches and related computational models become extremely unstable in the case of the very small amount of useful pixels that is typical in these conditions. Therefore, their behavior should be investigated more carefully. On the other hand, real-time emotion recognition at a distance may play a critical role in smart video surveillance, especially when controlling particular kinds of events, e.g., political meetings, to try to prevent adverse actions.This work compares facial expression recognition at a distance by: 1) a deep learning architecture based on state-of-the-art (SOTA) proposals, which exploits the whole images to autonomously learn the relevant embeddings; 2) a machine learning approach that relies on hand-crafted features, namely the facial landmarks preliminarily extracted using the popular Mediapipe framework. Instead of using either the complete sequence of frames or only the final still image of the expression, like current SOTA approaches, the two proposed methods are designed to use rich temporal information to identify three different stages of emotion. Expressions are time-split accordingly into four phases to better exploit their temporal-dependent dynamics. Experiments were conducted on the popular Extended Cohn-Kanade dataset (CK+). It was chosen for its wide use in related literature, and because it includes videos of facial expressions and not only still images. The results show that the approach relying on machine learning via hand-crafted features is more suitable for classifying the initial phases of the expression and does not decay in terms of accuracy when images are at a distance (only 0.08% of decay). On the contrary, deep learning not only has difficulties classifying the initial phases of the expressions but also suffers from relevant performance decay when considering images at a distance (52.68% accuracy decay).

Emotional freedom technique versus written exposure therapy versus waiting list for post-traumatic stress disorder: protocol for a randomised clinical MRI study

IntroductionThe emotional freedom technique (EFT) is an acupuncture-based psychotherapy that combines tapping on acupoints with cognitive reframing. EFT has been previously shown to have potential for treating post-traumatic stress disorder...

Predicting the effect of pharmacist’s communication with patients: medical communication analysis using facial responses

Abstract Objectives To provide patient-centred care, psychological knowledge, and skills are necessary for pharmaceutical communication. Acquisition of these communication skills is closely related to patient comprehension. Therefore, to improve pharmacist’s communication skills, pharmacist need to learn the characteristics of their medication instructions, such as posture, facial expressions, eye contact, nodding, and more. For the analysis of medical communication, there is a rating scale, functional analysis, and others. However, these methods may not match the actual emotions due to their analysis skills and the psychological stress of the patients. In this study, we examined the methods to evaluate patient-pharmacist communication using emotion recognition AI software, which recognises emotions from facial expressions. Methods With the cooperation of six simulated patients (SP) and eight pharmacists, we recorded the SP’s facial expressions during medication instruction. The facial expression video was analysed using emotion recognition AI, which can obtain emotion values (anger, contempt, disgust, fear, joy, sadness, surprise, and engagement). We compared the emotion of the extracted peaks with the feedback and calculated the emotion match rate. Key findings As a result, 33% of the emotions matched in the peak and feedback. This result indicates that emotion recognition AI cannot capture every feedback emotion. However, in joy, the result was not affected by engagement, and the match rate between peak and feedback was high. Conclusions In the future, emotion recognition AI will allow us to assess the effects of communication skills of the pharmacists on the psychological state of the patients more objectively and noninvasively.

On-road driver facial expression emotion recognition with parallel multi-verse optimizer (PMVO) and optical flow reconstruction for partial occlusion in internet of things (IoT)

In recent years, many large-scale information systems in the Internet of Things (IoT) can be converted into interdependent sensor networks, such as smart cities, smart medical systems, and industrial Internet systems. The successful application of Facial Expression Recognition in the IoT will make our algorithms faster, more convenient, lower overall costs, providing better business practices, and enhance sustainability. Facial Expression Recognition (FER)is essential to effectively communicate between human and machines. Video facial expression detection is a crucial component for gauging the driver's mood for driver assistance system. The emotions of the driver have a significant role in dictating the behavior of the driver, according to several studies, which can lead to disastrous car crashes. However, criteria affecting the identification of driver emotions with the right kind of monitoring include changes in stance, lighting, and occlusions. Based on the restoration of the blurred facial region, the Driver Facial Expression Emotion Recognition (DFEER) system was developed to address these issues. One of the first things to do when dealing with a sequence of blurred faces is to calculate the optical fluxes between the frames. After that, an optical flow reconstruction using a trained PMVO is performed to fix the occlusion-induced damage (Parallel Multi-Verse Optimizer). The major solutions are randomly split into groups of occluded and non-occluded optical flows using a parallel technique, and the groups discuss their findings after a set number of iterations. After optical flows have been rebuilt, they are used immediately in the classification phase of expression prediction. In this study, Very Deep Convolution Networks (VGGNet) proposed a method for recognising human emotions. Both the CK+ (Cohn-Kanade database) and the KMU-FED (Keimyung University Facial Expression of Drivers) databases are used to carry out the assessments of the classification model's efficacy. Accuracy, recall, precision, and f-measure the effectiveness of the suggested strategy is then evaluated using the findings.

Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos.

Depression is a common mental illness which has brought great harm to the individuals. With recent evidence that many objective physiological signals are associated with depression, automated detection of depression is urgent and important for the growing concern of mental illness. We investigate the problem of classifying depression by facial expressions, which may aid in online diagnosis and rehabilitation engineering of depression. In this work, We propose a weakly supervised learning approach employing multiple instance learning (MIL) on 150 videos data from 75 depressed and 75 healthy subjects. In addition, we present a novel MIL dual-stream aggregator that considers both the instance-level and the bag-level in order to emphasize the information with symptoms. Specifically, our method named ADDMIL uses max-pooling at the instance level to capture symptom information and further integrates the contribution of each instance at the bag level using attention weights. Our method achieves 74.7% accuracy and 74.5% recall on the collected dataset, which not only improves 10.1% accuracy and 9.8% recall over the baseline but also exceeds the best accuracy result of MIL-based method by 2.1%. Our work achieves results that are comparable to the state-of-the-art methods and demonstrates that multiple instance learning has great potential for depression classification. We present for the first time a weakly supervised learning approach in the detection of depression through raw facial expressions, which may provide a new framework for other psychiatric disorders detection methods.