Facial Display Research Articles

Cutting-Edge Methods in Visage Expression Recognition and Ocular Tracking for Analyzing Remote Video Interviews

This paper focuses to augment an in the flesh image and video processor, capacitated with an artificial intelligence (AI) arbitrator that can speculate a job interviewee’s behavioral adeptness according to the facial dictions or expressions. We put forward a sentiment analysis model using machine learning and CNN for the reinforcement of decision-making in the job interview process. This is bought up by histogram of oriented gradients and support vector machine (HOG-SVM) with the addition of convolutional neural network (CNN) recognition in real-time video recorded interviews. The goal of applying this technology, is to develop a method that could automatically decode a candidate’s behavior by his or her facial language (or micro expressions) based on the behavioral ecology view of facial displays (BECV) which is different from the classical perspective of recognizing emotional states. This paper dispenses a trailblazing technique of determining the performance of a candidate in a video interview. To do this, a delineation of the analysis of sentiments and eye tracking technique study is intricated in which the results can be processed on a single screen to select the right person for the hire. Key Words: Artificial Intelligence(AI), Histogram of Oriented Gradients(HOG), Support Vector Machine(SVM), Convolutional Neural Network(CNN), Behavioral Ecology View of Facial Displays, Eye Tracking Technique, AVI(Automatic Visual Inspection).

Binocular rivalry reveals differential face processing in congenital prosopagnosia

Congenital Prosopagnosia (CP) is an innate impairment in face perception with heterogeneous characteristics. It is still unclear if and to what degree holistic processing of faces is disrupted in CP. Such disruption would be expected to lead to a focus on local features of the face. In this study, we used binocular rivalry (BR) to implicitly measure face perception in conditions that favour holistic or local processing. The underlying assumption is that if stimulus saliency affects the perceptual dominance of a given stimulus in BR, one can deduce how salient a stimulus is for a given group (here: participants with and without CP) based on the measured perceptual dominance. A further open question is whether the deficit in face processing in CP extends to the processing of the facial display of emotions. In experiment 1, we compared predominance of upright and inverted faces displaying different emotions (fearful, happy, neutral) vs. houses between participants with CP (N = 21) and with normal face perception (N = 21). The results suggest that CP observers process emotions in faces automatically but rely more on local features than controls. The inversion of faces, which is supposed to disturb holistic processing, affected controls in a more pronounced way than participants with CP. In experiment 2, we introduced the Thatcher effect in BR by inverting the eye and mouth regions of the presented faces in the hope of further increasing the effect of face inversion. However, our expectations were not borne out by the results. Critically, both experiments showed that inversion effects were more pronounced in controls than in CP, suggesting that holistic face processing is less relevant in CP. We find BR to be a useful implicit test for assessing visual processing specificities in neurological participants.

“Anger? No, thank you. I don't mimic it”: how contextual modulation of facial display meaning impacts emotional mimicry

ABSTRACT Research indicates that emotional mimicry predominantly occurs in response to affiliative displays, such as happiness, while the mimicry of antagonistic displays, like anger, is seldom observed in social contexts. However, contextual factors, including the identity of the displayer (e.g. social similarity with the observer) and whose action triggered the emotional reaction (i.e. to whom display is directed), can modulate the meaning of the display. In two experiments, participants observed happiness, sadness, and anger expressed by individuals with similar or different social attitudes in response to actions from either a participant or another person. Results demonstrated that three manipulated factors – displayer social similarity, whose action caused an emotional display, and the type of emotional display – affected participants’ perception of the display. In turn, mimicry was predominantly observed in response to happiness (Experiments 1 and 2), to a lesser extent to sadness (Experiment 1), and not to anger. Furthermore, participants mimicked individuals who were more socially similar (Experiment 1), while whose action caused an emotional reaction did not influence mimicry. The findings suggest that when the context mitigates the meaning of negative or antagonistic facial displays, it does not necessarily increase the inclination to mimic them.

When a smile is still a conflict: Affective conflicts from emotional facial expressions of ingroup or outgroup members occur irrespective of the social interaction context

Facial expressions play a crucial role in human interactions. Typically, a positive (negative) expression evokes a congruent positive (negative) reaction within the observer. This congruent behavior is inverted, however, when the same positive (negative) expression is displayed by an outgroup member. Two approaches provide an explanation for this phenomenon. The social intentions account proposes underlying social messages within the facial display, whereas the processing conflict account assumes an affective conflict triggered by incongruent combinations of emotion and the affective connotation of group membership. In three experiments, we aimed at further substantiating the processing conflict account by separating the affective conflict from potential social intentions. For this, we created a new paradigm, in which the participant was an outside observer of a social interaction scene between two faces. Participants were required to respond to the emotional target person that could represent an ingroup or outgroup member. In all three experiments, irrespective of any social intention, responses were consistently affected by the group relation between participant and emotional target, i.e., the affective (in)congruency of the target seen by participants. These results further support the processing conflict account. The implications for the two theoretical accounts are discussed.

Exposure to emotions and postural stability in deaf youth

IntroductionThe population of deaf people is a heterogeneous group. Understanding the significant variables that affect motor, socio-emotional and cognitive development sets the direction and quality for subsequent developmental stages and provides the opportunity to select appropriate compensatory methods for deaf individuals.AimThe aim of the study was to verify the visual effect of the emotions of joy and anger on one of the dimensions of eye-hand coordination, i.e. postural stability among deaf and hearing youth.Material and methodsStudy subjects: 120 teenagers aged 13–17 years. The test group comprised 60 subjects with profound bilateral sensorineural hearing loss, while the control group comprised 60 hearing junior high school students. The study used a computerised stabilometric platform and photographs sourced from an affective image database Montreal Set of Facial Displays of Emotion.Results and discussionIn deaf youth, a switch in the stability of the body occurs, as compared to hearing youth. The results obtained indicate the need for the activation of cognitive, emotional and motor resources of deaf people with an area of multiple modalities taken into account.ConclusionsHuman development should be regarded as a continual process in relation to compensatory capacity.

Faking it deeply and universally? Media forms and epistemologies of artificial faces and emotions in Japanese and Euro-American contexts

Attempts to visually represent the human face in various media forms have a long history. Fairly recent examples can be found in techniques that are used in animated films and video games (performance capture) or in the controversial phenomenon of “deep fakes”. The article looks at the relationship between faces, the media forms that are used for their representation and simulation and the knowledge that is (re)produced about emotions and their communicative effects. It reveals a peculiar contradiction: the quest to create a synthetic digital face that would be universally recognisable in terms of emotional readability and response homogenises our understanding of emotion while disguising the challenging cultural barriers to this recognisability. The simulated face caters to a two-fold and hence paradoxical aesthetic aspiration that places it between uniqueness and universality. By elaborating on the somatic, affective dimension of communication via faces, the article compares culturally specific modes of producing and reading emotions. A Euro-American tradition of thought regards them as artefacts, constructed as antagonists of rationality, that can be controlled and objectively modelled and measured. The idea of the face as the locus of emotions, which can be represented in distinct expressions, contrasts with more ambiguous and relational meanings in East Asian, particularly Japanese contexts. The epistemological genealogy of the facial display of emotions, however, does not conform to a clear West/East divide and is at the same time highly interwoven with specific media forms of their representation. In order to deepen this dynamic understanding of artificial faces, the article looks at different examples of Japanese traditions and art forms. Examples such as portraiture and photography, the subtle illusions created in Nō theatre and the less subtle ones evoked by robotic faces shed a light on the complex meanings but also the affective reactions to artificial faces.

The relaxed open mouth is a true signal in dogs: demonstrating Tinbergen's ritualization process

Play fighting, the most iconic form of social play, is often punctuated by specific signals, such as the relaxed open mouth (ROM) display, limiting the risk of misunderstanding between playmates. Although there is general consensus that the ROM of dogs is a ritualized version of play biting, the empirical demonstration of the actual ritualization of ROM has been lacking. We videorecorded and analysed 118 playful sessions involving 24 Czechoslovakian wolfdogs (12 females; 12 males), which is a breed of domestic dog, Canis lupus familiaris, showing wolf-like behavioural traits. By using an integrated approach of different techniques (dog facial action coding system, an unsupervised cluster analysis and the Levenshtein distance), we empirically demonstrate that the ROM is intrinsically different from the play biting action in this breed of dog. Contrary to the play bite, during ROM, the recruitment of muscular action units for each facial display was more consistent, conspicuous and intra- and interindividually stereotyped. Moreover, a sequential analysis revealed that the ROM usually preceded playful offensive patterns, thus underlining the real metacommunicative function of the signal. Finally, by running a linear mixed model, we found that the most balanced sessions were punctuated by the most prolonged performance of ROM, thus revealing the efficiency of the facial signal in maintaining a balanced session. In conclusion, through the processes of formalization, simplification and emphasis, an ordinary precursor behaviour (i.e. play biting) has been taken out of context and transformed into an extraordinary, derived behaviour (i.e. ROM) specifically designed to attract receivers' attention and modulate playful social interactions in dogs.

Correspondence between Two Methods of Measuring Discrete Emotions: Self-report versus Machine-coded Facial Displays

Definitions and measures of discrete emotions are thorny issues in research on affect, resulting from the multitude of emotion theories and approaches to this concept; there is no gold standard for measures of discrete emotions. The utility of measurement methods should be compared across multiple perspectives to allow some degree of cumulativeness. This study reported emotion data collected from a college sample (N = 113), using seven professionally produced videos as stimuli messages. Fear, anger, sadness, disgust, and happiness were measured with the self-report method and by analyzing recorded facial expressions with FaceReader™, an FACS-based computer software that automatically analyzes facial expressions for discrete emotions. Multilevel modeling analyses demonstrated initial evidence for the correspondence between emotions measured with both methods and convergent and discriminant validity for FaceReader™ as a method of measuring discrete emotions.

The Effect of Facial Self-Resemblance on Emotional Mimicry

Social resemblance, like group membership or similar attitudes, increases the mimicry of the observed emotional facial display. In this study, we investigate whether facial self-resemblance (manipulated by computer morphing) modulates emotional mimicry in a similar manner. Participants watched dynamic expressions of faces that either did or did not resemble their own, while their facial muscle activity was measured using EMG. Additionally, after each presentation, respondents completed social evaluations of the faces they saw. The results show that self-resemblance evokes convergent facial reactions. More specifically, participants mimicked the happiness and, to a lesser extent, the anger of self-resembling faces. In turn, the happiness of non-resembling faces was less likely mimicked than in the case of self-resembling faces, while anger evoked a more divergent, smile-like response. Finally, we found that social evaluations were in general increased by happiness displays, but not influenced by resemblance. Overall, the study demonstrates an interesting and novel phenomenon, particularly that mimicry can be modified by relatively subtle cues of physical resemblance.

Humility Expression and its Effects on Moral Suasion: An Empirical Study of Ocasio-Cortez’s Communication

Abstract Humble leadership can be described as a positive psychological feature that allows leaders to admit their limitations, be open to new ideas, and give a voice to others while also recognizing their merits. The present study (n = 268 participants) explored the persuasive effects of a female politician communicating a humble stance by considering the role emotional displays at play (joy, calmness, sadness, and anger) when discussing a moral issue (hosting immigrants). The results revealed that the politician elicited positive emotions and evaluations of her competence and benevolence, especially when exhibiting a sad facial display, by contributing to the intention to accept the moral gist of her persuasive message. Overall, these ‘gendered’ effects are discussed in relation to the ‘political authenticity’, which can be perceived as high especially when the humble message is connected with several social reasons, as is the case with gender or social status.

Exerting control: the grammatical meaning of facial displays in signed languages

Abstract Signed languages employ finely articulated facial and head displays to express grammatical meanings such as mood and modality, complex propositions (conditionals, causal relations, complementation), information structure (topic, focus), assertions, content and yes/no questions, imperatives, and miratives. In this paper we examine two facial displays: an upper face display in which the eyebrows are pulled together called brow furrow, and a lower face display in which the corners of the mouth are turned down into a distinctive configuration that resembles a frown or upside-down U-shape. Our analysis employs Cognitive Grammar, specifically the control cycle and its manifestation in effective control and epistemic control. Our claim is that effective and epistemic control are associated with embodied actions. Prototypical physical effective control requires effortful activity and the forceful exertion of energy and is commonly correlated with upper face activity, often called the “face of effort.” The lower face display has been shown to be associated with epistemic indetermination, uncertainty, doubt, obviousness, and skepticism. We demonstrate that the control cycle unifies the diverse grammatical functions expressed by each facial display within a language, and that they express similar functions across a wide range of signed languages.

Automatic Recognition of Facial Displays of Unfelt Emotions

Humans modify their facial expressions in order to communicate their internal states and sometimes to mislead observers regarding their true emotional states. Evidence in experimental psychology shows that discriminative facial responses are short and subtle. This suggests that such behavior would be easier to distinguish when captured in high resolution at an increased frame rate. We are proposing SASE-FE, the first dataset of facial expressions that are either congruent or incongruent with underlying emotion states. We show that overall the problem of recognizing whether facial movements are expressions of authentic emotions or not can be successfully addressed by learning spatio-temporal representations of the data. For this purpose, we propose a method that aggregates features along fiducial trajectories in a deeply learnt space. Performance of the proposed model shows that on average, it is easier to distinguish among genuine facial expressions of emotion than among unfelt facial expressions of emotion and that certain emotion pairs such as contempt and disgust are more difficult to distinguish than the rest. Furthermore, the proposed methodology improves state of the art results on CK+ and OULU-CASIA datasets for video emotion recognition, and achieves competitive results when classifying facial action units on BP4D datase.

Facial displays in red-capped mangabeys (Cercocebus torquatus): Repertoire, social context, and potential intentionality.

Primate communication relies strongly on the visual modality, notably through the production of a wide range of expressive facial signals. We investigated here the facial display repertoire of a relatively little-studied cercopithecid species, red-capped mangabeys (Cercocebus torquatus), and questioned whether their facial displays were dependent on social contexts and accompanied by indices of intentionality. Although the dual intentional and emotional use of apes' facial expressions has recently been suggested, the question of whether monkeys produce facial expressions intentionally to communicate remains open. We described 6 facial displays produced by captive red-capped mangabeys in social contexts. They are based on movements of the mouth, eyebrows, and ears, possibly graded in intensity and produced independently or in combination. We showed that most of the facial displays were produced preferentially in specific social contexts and that repertoires varied with subjects' characteristics, highlighting the communicative function of these displays. Moreover, behavioral markers of intentionality commonly used in gestural studies were found to accompany the production of some of the facial signals observed. Particularly, playful "open mouth" appeared strongly associated with intentionality indices, as previously noticed in ape species. All other facial displays, except yawns, did not exhibit all defined intentional indices but were, at least, directed toward a recipient. Interestingly, yawns presented different variants of intensity associated presumably with different social functions. Altogether, these results emphasize the communicative function of red-capped mangabeys' facial displays and provide a basis for further research on their intentional communication. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Role of Sexual Selection in the Evolution of Facial Displays in Male Non-human Primates and Men

Sexual selection theory provides a framework through which some facial displays of male non-human primates can be investigated and understood. Here, we explore how both intra- and inter-sexual selection may influence facial morphology, physiology, and the behavior of male primates inhabiting diverse social and ecological environments. First, we will review how elements of the ecological environment, such as the spatio-temporal distribution of food, interacting with the diet of different species, shapes the social and mating systems of primates. In turn, these dynamics then influence how facial expressions, colors, and shapes are utilized in both the competition for and attraction of mates. We will focus on sexually dimorphic facial features that exhibit variation in their expression among males and that can be linked to differences in proxies of reproductive success. Facial displays can generally be divided into four types, each commonly associated with certain mating systems, social systems, and sexually selective pressures. Facial expressions, skin color, pelage, and variation in facial shape, can be involved in either mediating intra-sexual competition, mate choice, or a combination of both. Certain aspects of male non-human primate facial displays are likely to be shaped by processes of both intra- and inter-sexual selection. Accordingly, homologies and analogies between human and non-human primate facial displays can inform us of the processes of sexual selection that may have been operating throughout human evolution.

The structural and motivational role of the unique lip-flip movement in the gelada (Theropithecus gelada) facial display repertoire.

Human language represents an extreme form of communicative complexity. Primate facial display complexity, which depends upon facial mobility, can be used as a model for the study of the evolution of communicative complexity. The gelada (Theropithecus gelada) is the only primate that can produce a lip-flip eversion. This study investigates the role of the lip-flip relative to the bared-teeth display to understand its role in generating communicative complexity. We reviewed videos of gelada social interactions. We utilized the facial action coding system (FACS) to define structural component action units (AUs) of each display. We inferred display motivation from the behaviors of the display sender. The lip-flip was used only in combination with the essential AUs of the bared-teeth display, serving as an optional structural element added to produce a structural variant. Both the bared-teeth display with and without a lip-flip occurred most frequently with nonaggressive, submissive behaviors. The lip-flip was more frequently preceded by approach than the bared-teeth display, especially in males. The lip-flip was also present in the majority of structurally blended facial displays though the motivation of the non-lip-flip parent display often dominated. The lip-flip may potentially function as an indicator of benign intent after an approach or as an intensifying component of nonaggressive intent. Adaptations to increase facial mobility in geladas via facilitating the lip-flip may promote increased communicative complexity through increased conspicuousness and motivational signaling specification or intensification.

Perspective-taking influences attentional deployment towards facial expressions of pain: an eye-tracking study.

Empathetic perspective-taking (PT) may be critical in modulating attention and associated responses to another's pain. However, the differential effects of imagining oneself to be in the pain sufferer's situation ("Self-perspective") or imagining the negative impacts on the pain sufferer's experience ("Other-perspective") on attention have not been studied. The effects of observer PT (Self vs Other) and level of facial pain expressiveness (FPE) upon attention to another person's pain was investigated. Fifty-two adults were assigned to 1 of 3 PT conditions; they were instructed to view pairs of pain expressions and neutral faces and either (1) consider their own feelings (Self-perspective), (2) consider the feelings of the person in the picture (Other-perspective), or (3) received no further instructions (Control). Eye movements provided indices of early (probability and duration of first fixation) and later (total gaze duration) attentional deployment. Pain faces were more likely to be fixated upon first. A significant first fixation duration bias towards pain was observed, which increased with increasing levels of FPE, and was higher in the Self-PT than the Control condition. The proportion of total gaze duration on pain faces was higher in both experimental conditions than the Control condition. This effect was moderated by FPE in the Self-PT condition; there was a significant increase from low to high FPE. When observers attend to another's facial display of pain, top-down influences (such as PT) and bottom-up influences (such as sufferer's FPE) interact to control deployment and maintenance of attention.

‘Too humble and sad’: The effect of humility and emotional display when a politician talks about a moral issue

Humble leadership can be defined as a positive psychological feature that allows for admitting one’s own limitations, being open to new ideas and giving a voice to, and recognizing the merits of, others. Humility has been associated with positive emotions, like gratitude and elevation, however it has been neglected in the political domain, seen on the contrary as the management of power. The present experimental study was aimed at exploring the possible persuasive effect of a humble politician by considering the role played by emotional display (joy, calm, sadness and anger) when the politician discusses an ethical issue (such as hosting immigration). Contrary to our expectations, humble communication elicited negative emotions and evaluations of the politician, especially when the politician exhibited a sad facial display; further, it induced in voters a state of personal distress towards immigrants, and thus a negative association with a hosting orientation.

Does dynamic visual information in talking faces influence the perceptual restoration of phonemes?

It is well-established that the perception of speech can be highly influenced by visible articulatory information. Recently, Irwin et al. (2017) demonstrated a robust effect in which visual speech cues perceptually “restore” a speech sound that has been acoustically weakened. Here we investigated the nature of the visual information that elicits this perceptual illusion. To accomplish this, we utilized an oddball paradigm in which perceivers were presented with acoustic /ba/ (the more frequently occurring standard stimulus) and /a/ tokens (the infrequently presented deviant stimulus). The acoustic tokens were dubbed with three types of video tokens: (1) A full face articulating /ba/; (2) the same face articulating /ba/ but with the oral-facial region pixelated; or (3) a point-light facial display of the produced /ba/ that depicted the isolated kinematics of the visible lip movements. Results indicated that perceivers showed visual phonemic restoration (reduced accuracy in detecting deviant /a/) in the presence of the natural talking face, but not in the presence of either the pixelated or schematic (point-light) faces. These findings suggest that the extracted kinematic information may not be sufficient to elicit the restoration effect, or that isolated kinematic cues do not integrate with acoustic speech in a robust manner.It is well-established that the perception of speech can be highly influenced by visible articulatory information. Recently, Irwin et al. (2017) demonstrated a robust effect in which visual speech cues perceptually “restore” a speech sound that has been acoustically weakened. Here we investigated the nature of the visual information that elicits this perceptual illusion. To accomplish this, we utilized an oddball paradigm in which perceivers were presented with acoustic /ba/ (the more frequently occurring standard stimulus) and /a/ tokens (the infrequently presented deviant stimulus). The acoustic tokens were dubbed with three types of video tokens: (1) A full face articulating /ba/; (2) the same face articulating /ba/ but with the oral-facial region pixelated; or (3) a point-light facial display of the produced /ba/ that depicted the isolated kinematics of the visible lip movements. Results indicated that perceivers showed visual phonemic restoration (reduced accuracy in detecting deviant /a/) in the pre...

Facing exclusion and smiling through the pain: positive emotion expression during interpersonal ostracism.

Background: The present research compares smiling and emotion expression generally to other indicators of negative intra- and interpersonal reactions to ostracism, and in particular negative reactions towards the ostracizers.Methods: A total of N=143 participants (n=55 in Experiment 1 and n=88 in Experiment 2) were ostracized from a web-conference by two other individuals. Facial expressions of participants during the exclusion period were coded using EMFACS and compared to self-reported reactions to ostracism and the sources of the ostracism.Results: Ostracized individuals showed significant levels of both Duchenne (genuine) and non-Duchenne (social) smiling, despite finding ostracism highly aversive, reporting more negative attitudes towards the ostracizing confederates, and (in Experiment 2) higher levels of negative affect. Experiment 2 showed evidence of a self-regulation and display management function of smiling during ostracism in that participants who exhibited more Duchenne smiling during their exclusion also reported higher levels of positive emotion after the ostracism, and were also rated by a group of judges as experiencing more amusement at their exclusion.Conclusion: The web conferencing paradigm used in this study provides an ecologically valid method to study the management of expressive behavior during aversive interpersonal experiences, adding to the existing evidence of facial display management during other types of distressing experience.

Facial Affect and Interpersonal Affiliation: Displays of Emotion During Relationship Formation in Social Anxiety Disorder.

Social anxiety disorder (SAD) often involves difficulty developing relationships. Facial expressions are important in relationship formation, but data are limited regarding facial display production among persons with SAD during social interaction. The current study compared facial displays of individuals diagnosed with SAD (n=41) to control participants (n=24) as they interacted with a confederate; confederates and observers then rated their desire for future interaction with participants. Automated software used the Facial Action Coding System (FACS; Ekman & Friesen, 1978) to classify displays. During portions of the interaction that involved listening to partners, the SAD group smiled less frequently and less intensely than controls, and lower smiling was associated with others' lower desire for future interaction with participants. Diminished positive facial affect in response to interaction partners may disrupt relationship formation in SAD and may serve as an effective treatment target.