Facial Action Research Articles

Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond

Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space, or parameter transfer. To provide sufficient learning support, modern MTL uses annotated data with full, or sufficiently large overlap across tasks, i.e., each input sample is annotated for all, or most of the tasks. However, collecting such annotations is prohibitive in many real applications, and cannot benefit from datasets available for individual tasks. In this work, we challenge this setup and show that MTL can be successful with classification tasks with little, or non-overlapping annotations, or when there is big discrepancy in the size of labeled data per task. We explore task-relatedness for co-annotation and co-training, and propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching. To demonstrate the general applicability of our method, we conducted diverse case studies in the domains of affective computing, face recognition, species recognition, and shopping item classification using nine datasets. Our large-scale study of affective tasks for basic expression recognition and facial action unit detection illustrates that our approach is network agnostic and brings large performance improvements compared to the state-of-the-art in both tasks and across all studied databases. In all case studies, we show that co-training via task-relatedness is advantageous and prevents negative transfer (which occurs when MT model's performance is worse than that of at least one single-task model).

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Although deep generative models have greatly improved one-shot video-driven talking head generation, few studies address fine-grained controllable facial expression editing, which is crucial for practical applications. Existing methods rely on a fixed set of predefined discrete emotion labels or simply copy expressions from input videos. This is limiting as expressions are complex, and methods using only emotion labels cannot generate fine-grained, accurate or mixed expressions. Generating talking head video with precise expressions is also difficult using 3D model-based approaches, as 3DMM only models facial movements and tends to produce deviations. In this paper, we propose a novel framework enabling fine-grained facial expression editing in talking face generation. Our goal is to achieve expression control by manipulating the intensities of individual facial Action Units (AUs) or groups. First, compared with existing methods which decouple the face into pose and expression, we propose a disentanglement scheme to isolates three components from the human face, namely, appearance, pose, and expression. Second, we propose to use input AUs to control muscle group intensities in the generated face, and integrate the AUs features with the disentangled expression latent code. Finally, we present a self-supervised training strategy with well-designed constraints. Experiments show our method achieves fine-grained expression control, produces high-quality talking head videos and outperforms baseline methods.

Exploring the impact of computer-mediated emotional interactions on human facial and physiological responses

Remote communication has become pervasive, yet its impact on human emotions, empathy, and physiological responses remains unclear. This study addresses this gap by investigating how emotional video-mediated conversations differ from face-to-face interactions, focusing on behavioral and physiological responses. We create a multimodal dataset of Electrodermal Activity (EDA) signals, Photoplethysmography (PPG) signals, and facial videos from two people conversing about emotional topics in face-to-face and remote video-mediated conditions. We use a series of repeated measures ANOVA with aligned rank transform to compare face-to-face to remote conversation in terms of heart rate activity, electrodermal activity, and facial action units. We also explore how subjective empathy between people varies in these two conditions. Our findings reveal significant differences in physiological responses between face-to-face and remote conversation and variations in perceived empathy based on interaction setting, highlighting the nuanced influence of communication channels. We also show that we can recognize emotions more accurately when we pre-train a random forest classifier with one condition’s data (an increase of 20% to 45% for various modalities). Finally, we discuss the research findings and limitations and offer insights for optimizing human–computer interaction and understanding human emotional responses in an increasingly tech-mediated world.

Decoding pain through facial expressions: a study of patients with migraine

BackgroundThe present study used the Facial Action Coding System (FACS) to analyse changes in facial activities in individuals with migraine during resting conditions to determine the potential of facial expressions to convey information about pain during headache episodes.MethodsFacial activity was recorded in calm and resting conditions by using a camera for both healthy controls (HC) and patients with episodic migraine (EM) and chronic migraine (CM). The FACS was employed to analyse the collected facial images, and intensity scores for each of the 20 action units (AUs) representing expressions were generated. The groups and headache pain conditions were then examined for each AU.ResultsThe study involved 304 participants, that is, 46 HCs, 174 patients with EM, and 84 patients with CM. Elevated headache pain levels were associated with increased lid tightener activity and reduced mouth stretch. In the CM group, moderate to severe headache attacks exhibited decreased activation in the mouth stretch, alongside increased activation in the lid tightener, nose wrinkle, and cheek raiser, compared to mild headache attacks (all corrected p < 0.05). Notably, lid tightener activation was positively correlated with the Numeric Rating Scale (NRS) level of headache (p = 0.012). Moreover, the lip corner depressor was identified to be indicative of emotional depression severity (p < 0.001).ConclusionFacial expressions, particularly lid tightener actions, served as inherent indicators of headache intensity in individuals with migraine, even during resting conditions. This indicates that the proposed approach holds promise for providing a subjective evaluation of headaches, offering the benefits of real-time assessment and convenience for patients with migraine.

In the eye of the promoter? How faculty ratings of attractiveness matter for junior academic careers

We investigate whether junior researchers’ attractiveness matters for obtaining an academic job post-PhD in a specific field within the social sciences, using faculty ratings on photographs along with data on publications, quality of PhD-granting university and employment type. The use of faculty ratings rather than external rating platforms allows for a closer match to a real-life academic setting. We find a significant association between faculty-perceived attractiveness and the probability of junior female researchers being in academic positions. A one standard deviation increase in the faculty attractiveness rating is associated with an increase in female junior researchers’ academic job holding by 25 % of the mean. This association is equally strong for the full sample of females and a seasoned subsample of females, observed at least 5 years post-PhD. We show that faculty ratings are different from general perceptions, the latter being irrelevant for junior researchers’ academic job holding. We also find that research productivity or access to high-quality coauthorship networks cannot explain the association. Collegiality, inferred using facial action coding of the photographs, is associated with the probability of females being in academia, but does not alter the association between faculty perceived attractiveness and females’ chances of an academic job post-PhD.

Comparing face-tracking action units with electromyography during speech

Video face-tracking software, such as OpenFace 2.0, can be used to make inferences about facial muscle activation [Baltrušaitis et al. IEEE 13, 59–66 (2018)]. However, the accuracy of these inferences based on the facial action units (FAUs) calculated by OpenFace 2.0 compared to corresponding muscle activity during speech is unclear. A previous study investigated muscle activation when both smile and speech occur simultaneously, focusing on the zygomaticus major (ZM) muscle [Liu et al. ISSP, 130–133 (2021)], but only presented data for a single speaker and did not compare FAU and EMG results. The present study compares OpenFace 2.0 action units with surface electromyography (EMG) data during speech in order to assess the validity of these inferences about facial muscle activation. We compare ZM activity and the lip corner puller FAU intensity results from a dataset collected for the previously mentioned [Liu et al. 2021] study. Data include four speakers producing read speech in smile conditions; 2 s will be extracted from EMG and FAU data before and after each utterance. Results will be reported on the relative accuracy of the FAU and EMG data. Implications will be discussed for speech communication research. [Work supported by NSERC.]

Drowsiness and Crash Detection Mobile Application for Vehicle’s Safety

Detecting road accidents promptly is crucial for minimizing casualties and property damage worldwide. The proposed system, comprising hardware and a mobile application, automatically identifies and reports accidents to emergency services. It also employs a facial recognition system to detect driver drowsiness, enhancing accident prevention measures. By leveraging sensor technologies, cellular networks, and advanced detection algorithms, the proposed system analyzes data from accelerometers, Global System for Mobile Communication (GSM), and Global Positioning System (GPS) sensors. Originally designed for vehicles, it can be easily adapted for deployment in various settings such as factories and construction sites with minor adjustments. The system continuously monitors the driver's facial expressions and activities using sensors. When drowsiness is detected, it activates a buzzer, and in the event of a crash, it alerts the driver to prevent false alarms while simultaneously notifying the rescue center if a genuine crash has occurred. This integrated approach enhances safety and optimizes emergency response efforts. The Arduino microcontroller, equipped with an accelerometer, identifies sudden changes in motion like acceleration and rotation to assess impacts against predefined thresholds. Furthermore, GPS functionality accurately determines the vehicle's location at the time of the accident, while GSM enables seamless communication with rescue centers through notifications.

Can Perceivers Differentiate Intense Facial Expressions? Eye Movement Patterns.

Recent research on intense real-life faces has shown that although there was an objective difference in facial activities between intense winning faces and losing faces, viewers failed to differentiate the valence of such expressions. In the present study, we explored whether participants could perceive the difference between intense positive facial expressions and intense negative facial expressions in a forced-choice response task using eye-tracking techniques. Behavioral results showed that the recognition accuracy rate for intense facial expressions was significantly above the chance level. For eye-movement patterns, the results indicated that participants gazed more and longer toward the upper facial region (eyes) than the lower region (mouth) for intense losing faces. However, the gaze patterns were reversed for intense winning faces. The eye movement pattern for successful differentiation trials did not differ from failed differentiation trials. These findings provided preliminary evidence that viewers can utilize intense facial expression information and perceive the difference between intense winning faces and intense losing faces produced by tennis players in a forced-choice response task.

Soft electrodes for simultaneous bio-potential and bio-impedance study of the face

The human body’s vascular system is a finely regulated network: blood vessels can change in shape (i.e. constrict, or dilate), their elastic response may shift and they may undergo temporary and partial blockages due to pressure applied by skeletal muscles in their immediate vicinity. Simultaneous measurement of muscle activation and the corresponding changes in vessel diameter, in particular at anatomical regions such as the face, is challenging, and how muscle activation constricts blood vessels has been experimentally largely overlooked. Here we report on a new electronic skin technology for facial investigations to address this challenge. The technology consists of screen-printed dry carbon electrodes on soft polyurethane substrate. Two dry electrode arrays were placed on the face: One array for bio-potential measurements to capture muscle activity and a second array for bio-impedance. For the bio-potential signals, independent component analysis (ICA) was used to differentiate different muscle activations. Four-contact bio-impedance measurements were used to extract changes (related to artery volume change), as well as beats per minute (BPM). We performed concurrent bio-potential and bio-impedance measurements in the face. From the simultaneous measurements we successfully captured fluctuations in the superficial temporal artery diameter in response to facial muscle activity, which ultimately changes blood flow. The observed changes in the face, following muscle activation, were consistent with measurements in the forearm and were found to be notably more intricate. Both at the arm and the face, a clear increase in the baseline impedance was recorded during muscle activation (artery narrowing), while the impedance changes signifying the pulse had a clear repetitive trend only at the forearm. These results reveal the direct connection between muscle activation and the blood vessels in their vicinity and start to unveil the complex mechanisms through which facial muscles might modulate blood flow and possibly affect human physiology.

Facial electromyography during exercise using soft electrode array: A feasibility study.

The use of wearable sensors for real-time monitoring of exercise-related measures has been extensively studied in recent years (e.g., performance enhancement, optimizing athlete's training, and preventing injuries). Surface electromyography (sEMG), which measures muscle activity, is a widely researched technology in exercise monitoring. However, due to their cumbersome nature, traditional sEMG electrodes are limited. In particular, facial EMG (fEMG) studies in physical training have been limited, with some scarce evidence suggesting that fEMG may be used to monitor exercise-related measurements. Altogether, sEMG recordings from facial muscles in the context of exercise have been examined relatively inadequately. In this feasibility study, we assessed the ability of a new wearable sEMG technology to measure facial muscle activity during exercise. Six young, healthy, and recreationally active participants (5 females), performed an incremental cycling exercise test until exhaustion, while facial sEMG and vastus lateralis (VL) EMG were measured. Facial sEMG signals from both natural expressions and voluntary smiles were successfully recorded. Stable recordings and high-resolution facial muscle activity mapping were achieved during different exercise intensities until exhaustion. Strong correlations were found between VL and multiple facial muscles' activity during voluntary smiles during exercise, with statistically significant coefficients ranging from 0.80 to 0.95 (p<0.05). This study demonstrates the feasibility of monitoring facial muscle activity during exercise, with potential implications for sports medicine and exercise physiology, particularly in monitoring exercise intensity and fatigue.

Development of PainFace software to simplify, standardize, and scale up mouse grimace analyses.

Facial grimacing is used to quantify spontaneous pain in mice and other mammals, but scoring relies on humans with different levels of proficiency. Here, we developed a cloud-based software platform called PainFace ( http://painface.net ) that uses machine learning to detect 4 facial action units of the mouse grimace scale (orbitals, nose, ears, whiskers) and score facial grimaces of black-coated C57BL/6 male and female mice on a 0 to 8 scale. Platform accuracy was validated in 2 different laboratories, with 3 conditions that evoke grimacing-laparotomy surgery, bilateral hindpaw injection of carrageenan, and intraplantar injection of formalin. PainFace can generate up to 1 grimace score per second from a standard 30 frames/s video, making it possible to quantify facial grimacing over time, and operates at a speed that scales with computing power. By analyzing the frequency distribution of grimace scores, we found that mice spent 7x more time in a "high grimace" state following laparotomy surgery relative to sham surgery controls. Our study shows that PainFace reproducibly quantifies facial grimaces indicative of nonevoked spontaneous pain and enables laboratories to standardize and scale-up facial grimace analyses.

Zygomaticus activation through facial neuromuscular electrical stimulation (fNMES) induces happiness perception in ambiguous facial expressions and affects neural correlates of face processing.

The role of facial feedback in facial emotion recognition remains controversial, partly due to limitations of the existing methods to manipulate the activation of facial muscles, such as voluntary posing of facial expressions or holding a pen in the mouth. These procedures are indeed limited in their control over which muscles are (de)activated when and to what degree. To overcome these limitations and investigate in a more controlled way if facial emotion recognition is modulated by one's facial muscle activity, we used computer-controlled facial neuromuscular electrical stimulation (fNMES). In a pre-registered EEG experiment, ambiguous facial expressions were categorised as happy or sad by 47 participants. In half of the trials, weak smiling was induced through fNMES delivered to the bilateral Zygomaticus Major muscle for 500 ms. The likelihood of categorising ambiguous facial expressions as happy was significantly increased with fNMES, as shown with frequentist and Bayesian linear mixed models. Further, fNMES resulted in a reduction of P1, N170 and LPP amplitudes. These findings suggest that fNMES-induced facial feedback can bias facial emotion recognition and modulate the neural correlates of face processing. We conclude that fNMES has potential as a tool for studying the effects of facial feedback.

Does it bug you eating bugs? The emotions of anthropoentomophagy

PurposeThe use of insects as food is a proposed solution for the increased demand for food worldwide, but it lacks acceptance because of restrictive emotional factors. This article aims to understand better customers' emotions’ role in considering and consuming insect-based food.Design/methodology/approachTo assess their acceptance, an experiment was developed with 38 participants living in Portugal to identify how people feel when consuming processed insect bars compared to cereal bars (of equal flavour). A video was recorded “before”, “during” and “after” the consumption of such foods, and the triggered emotions and affective states were identified using the Facial Action Coding System (FACS) and the circumplex model of affect, respectively. After consumption, the Self-Assessment Manikin (SAM) was asked to be completed.FindingsIt was observed that the valence and arousal of the emotions and affective states triggered during consumption were higher in the insect bar than in the cereal bar. Its consumption resulted in surprise and a positive evaluation. Processed insect-based foods may result in a potentially increased acceptance of this new food alternative in the market.Originality/valuePrior studies briefly identified disgust as a primary emotion activated by insect-based food. The current research deeply studied emotional responses to insect-based processed foods in the Western world using the dimensional emotional models. This study offers arguments for the insect-based food industry to invest in processed food justified by its potential for acceptance. In addition, it motivates further research focused on other insect-based products (e.g. non-processed ones).

Sex differences in facial expressions of pain: results from a combined sample.

Facial expressions of pain play an important role in pain diagnostics and social interactions. Given the prominent impact of sex on various aspects of pain, it is not surprising that sex differences have also been explored regarding facial expressions of pain; however, with inconclusive findings. We aim to further investigate sex differences in facial expressions of pain by using a large, combined sample to maximize statistical power. Data from 7 previous studies of our group were merged, combining in total the data of 392 participants (male: 192, female: 200). All participants received phasic heat pain, with intensities being tailored to the individual pain threshold. Pain intensity ratings were assessed, and facial responses were manually analyzed using the Facial Action Coding. To compare facial and subjective responses between sexes, linear mixed-effects models were used, with study ID as a random effect. We found significant sex differences in facial responses, with females showing elevated facial responses to pain, although they received lower physical heat intensities (women had lower pain thresholds). In contrast, pain intensity ratings did not differ between sexes. Additionally, facial and subjective responses to pain were significantly associated across sexes, with females showing slightly stronger associations. Although variations in facial expressions of pain are very large even within each sex, our findings demonstrate that women facially communicate pain more intensively and with a better match to their subjective experience compared with men. This indicates that women might be better in using facial communication of pain in an intensity-discriminative manner.

Infants' facial electromyographic responses to the sight of emotional interpersonal touch.

Adult studies have shown that observed interpersonal touch provides crucial information about others' emotional states. Yet, despite the unique communicative function of touch during development, very little is known about infants' sensitivity to the emotional valence of observed touches. To investigate this issue, we measured facial electromyographic (EMG) activity in response to positive (caress) and negative (scratches) observed touches in a sample of 11-month-old infants. Facial EMG activity was measured over the zygomaticus major (ZM) and corrugator supercilii muscles, respectively involved in positive (i.e., smiling) and negative (i.e., frowning) facial expressions. Results have shown distinct activations of the ZM during the observation of scratches and caresses. In particular, significantly greater activation of the ZM (smiling muscle) emerged specifically in response to the observation of caresses compared to scratches. Our finding suggests that, in infancy, observed affective touches can evoke emotional facial reactions.

Facial disgust in response to touches, smells, and tastes.

Disgust serves to defend the body from the entry of toxins and disease. Central to this function is a strong relationship with the proximate senses of smell, taste, and touch. Theory suggests that distinct and reflexive facial movements should be evoked by gustatory and olfactory disgusts, serving to impede bodily entry. While this hypothesis has received some support from facial recognition studies, whether smell and taste disgusts actually produce distinct facial responses, is unknown. Moreover, there has been no assessment of the facial response evoked by contact with disgusting objects. To address these issues, this study compared facial responses to touch, smell, and taste disgusts. Sixty-four participants were asked to touch, smell, and taste disgust-evoking and neutral control stimuli, and rate them on disgust, on two occasions-first, while they were video recorded and second, with facial electromyography (EMG) applied (measuring levator labii and corrugator supercilii activity). Videos were coded for facial expressions by humans and for facial action units (FAUs) by machines. Self-report data confirmed the disgust stimuli as highly disgusting. Comparison of the overall pattern of FAUs evoked by touch, smell, and taste disgusts, indicated two distinct facial disgusts for the proximate senses-a chemosensory and a tactile-disgust face. The nose wrinkle and upper lip raise were central to all facial disgusts, indicating their centrality to the disgust face. Several facial disgusts appear to exist, each with different functional goals. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Detecting Facial Action Units From Global-Local Fine-Grained Expressions

Since Facial Action Unit (AU) annotations require domain expertise, common AU datasets only contain a limited number of subjects. As a result, a crucial challenge for AU detection is addressing identity overfitting. We find that AUs and facial expressions are highly associated, and existing facial expression datasets often contain a large number of identities. In this paper, we aim to utilize the expression datasets without AU labels to facilitate AU detection. Specifically, we develop a novel AU detection framework aided by the Global-Local facial Expressions Embedding, dubbed GLEE-Net. Our GLEE-Net consists of three branches to extract identity-independent expression features for AU detection. We introduce a global branch for modeling the overall facial expression while eliminating the impacts of identities. We also design a local branch focusing on specific local face regions. The combined output of global and local branches is firstly pre-trained on an expression dataset as an identity-independent expression embedding, and then finetuned on AU datasets. Therefore, we significantly alleviate the issue of limited identities. Furthermore, we introduce a 3D global branch that extracts expression coefficients through 3D face reconstruction to consolidate 2D AU descriptions. Finally, a Transformer-based multi-label classifier is employed to fuse all the representations for AU detection. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art on the widely-used DISFA, BP4D and BP4D+ datasets.

Primate socio-ecology shapes the evolution of distinctive facial repertoires.

Primate facial musculature enables a wide variety of movements during bouts of communication, but how these movements contribute to signal construction and repertoire size is unclear. The facial mobility hypothesis suggests that morphological constraints shape the evolution of facial repertoires: species with higher facial mobility will produce larger and more complex repertoires. In contrast, the socio-ecological complexity hypothesis suggests that social needs shape the evolution of facial repertoires: as social complexity increases, so does communicative repertoire size. We tested these two hypotheses by comparing chimpanzees (Pan troglodytes) and gibbons (family Hylobatidae), two distantly related apes who vary in their facial mobility and social organization. While gibbons have higher facial mobility than chimpanzees, chimpanzees live in more complex social groups than gibbons. We compared the morphology and complexity of facial repertoires for both apes using Facial Action Coding Systems designed for chimpanzees and gibbons. Our comparisons were made at the level of individual muscle movements (action units [AUs]) and the level of muscle movement combinations (AU combinations). Our results show that the chimpanzee facial signaling repertoire was larger and more complex than gibbons, consistent with the socio-ecological complexity hypothesis. On average, chimpanzees produced AU combinations consisting of more morphologically distinct AUs than gibbons. Moreover, chimpanzees also produced more morphologically distinct AU combinations than gibbons, even when focusing exclusively on AUs present in both apes. Therefore, our results suggest that socio-ecological factors were more important than anatomical ones to the evolution of facial signaling repertoires in chimpanzees and gibbons. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Effect of distance education courses held in different environments on emotions of the instructor.

In this study, the emotional states of the instructors who teach by distance education and the effect of the environment in which they give the lesson on their emotions were examined. Computer-aided "Facial Action Coding System" method was used to measure emotion values from facial images. Through the software developed by the researchers using the Microsoft Face Recognition API, 43292 facial images taken from five trainers during their training were analysed and seven basic emotions representing facial expressions were obtained numerically. As a result of the analysis, it was found that the emotions of the instructors that can be described as negative in the lessons held in the e-studio environment generally increased at the beginning of the lesson, decreased in the following minutes and increased again at the end of the lesson; On the other hand, it was determined that positive emotions decreased at the beginning of the lesson and increased later. In the home environment, while the emotions that can be described as negative at the beginning decreased, positive emotions increased. A significant difference was determined between home and e-studio environment in all emotions except anger. One of the emotions with a difference in value between the two environments is happiness, and it has been determined that happiness has higher values in the home environment. It has been determined that other emotions are experienced more in the e-studio environment. It is thought that the results of the study will contribute to the mental states of the instructors who teach through distance education and to the efficiency of distance education.

Automated facial expression measurement in a longitudinal sample of 4- and 8-month-olds: Baby FaceReader 9 and manual coding of affective expressions

Facial expressions are among the earliest behaviors infants use to express emotional states, and are crucial to preverbal social interaction. Manual coding of infant facial expressions, however, is laborious and poses limitations to replicability. Recent developments in computer vision have advanced automated facial expression analyses in adults, providing reproducible results at lower time investment. Baby FaceReader 9 is commercially available software for automated measurement of infant facial expressions, but has received little validation. We compared Baby FaceReader 9 output to manual micro-coding of positive, negative, or neutral facial expressions in a longitudinal dataset of 58 infants at 4 and 8 months of age during naturalistic face-to-face interactions with the mother, father, and an unfamiliar adult. Baby FaceReader 9’s global emotional valence formula yielded reasonable classification accuracy (AUC = .81) for discriminating manually coded positive from negative/neutral facial expressions; however, the discrimination of negative from neutral facial expressions was not reliable (AUC = .58). Automatically detected a priori action unit (AU) configurations for distinguishing positive from negative facial expressions based on existing literature were also not reliable. A parsimonious approach using only automatically detected smiling (AU12) yielded good performance for discriminating positive from negative/neutral facial expressions (AUC = .86). Likewise, automatically detected brow lowering (AU3+AU4) reliably distinguished neutral from negative facial expressions (AUC = .79). These results provide initial support for the use of selected automatically detected individual facial actions to index positive and negative affect in young infants, but shed doubt on the accuracy of complex a priori formulas.