Emotion Inference Research Articles

Automatic Estimation of Action Unit Intensities and Inference of Emotional Appraisals

The development of a two-stage approach for appraisal inference from automatically detected Action Unit (AUs) intensities in recordings of human faces is described. AU intensity estimation is based on a hybrid approach fusing information from individually fitted mesh model of the faces and texture information. Evaluation results for two datasets and a comparison against a state of the art system are provided. In the second stage, the emotional appraisals novelty, valence and control are predicted from estimated AU intensities by linear regressions. Prediction performance is evaluated based on face recordings from a market research study, which were rated by human observers in terms of perceived appraisals. Predictions of valence and control from automatically estimated AU intensities closely match those obtained from manually coded AUs in terms of agreement with human observers, while novelty predictions lag somewhat behind. Overall, results highlight the flexibility and interpretability of a two-stage approach to emotion inference.

Oh … so close! Children's close counterfactual reasoning and emotion inferences.

How we feel about an outcome often depends on how close an alternative outcome was to occurring. In four experiments, we investigated whether predominantly White, middle-class, Canadian children (N = 425, Experiments 1-3) and American adults (N = 227, Experiment 4) consider close counterfactual alternatives when inferring other people's emotions. In Experiment 1, 6-year-olds (but not 4- and 5-year-olds) inferred that an agent would feel sadder about winning a mediocre prize if she later found out that a more attractive one could have easily been won. However, children of all ages failed to judge whether the better outcome could have easily happened. In Experiment 2, when 5- and 6-year-olds knew the locations of the prizes beforehand, they inferred that an agent would be equally happy about winning a mediocre prize, regardless if she almost won a better prize or not. Again, they did not recognize when the better outcome was a close counterfactual possibility. In Experiment 3, we included extra cues to the closeness of the alternative and both 5- and 6-year-olds inferred that she would feel sadder about winning a mediocre prize, and 6-year-olds acknowledged that the attractive prize was a close counterfactual alternative. In Experiment 4, adults considered close counterfactuals when inferring emotions. Our findings suggest that close counterfactuals influence children's emotion inferences before they become able to acknowledge their closeness. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Evidence against emotion inference deficits in children with ADHD.

Inconsistent evidence suggests that pediatric attention deficit/hyperactivity disorder (ADHD) may be associated with impairments in the ability to use context clues to infer the emotion states of others. However, the evidence base for these impairments is comprised of data from laboratory-based tests of emotion inference that may be confounded by demands on nonaffective cognitive processes that have been linked with ADHD. The current study builds on our previous study of facial affect recognition to address this limitation and investigate a potential mechanism underlying children's ability to infer emotion state from context clues. To do so, we used a fully crossed, counterbalanced experimental design that systematically manipulated emotion inference and working memory demands in 77 carefully phenotyped children ages 8-13 (Mage = 10.46, SD = 1.54; 66% Caucasian/Non-Hispanic; 42% female) with ADHD (n = 42) and without ADHD (n = 35). Results of Bayesian mixed-model ANOVAs indicated that using context clues to infer the emotion state of others competed for neurocognitive resources with the processes involved in rehearsing/maintaining information within working memory (BF₁₀ = 1.57 × 10¹⁹, d = 0.72). Importantly, there was significant evidence against the critical Group × Condition interaction for response times (BF₀₁ = 4.93), and no significant evidence for this interaction for accuracy (BF₀₁ = 2.40). In other words, children with ADHD do not infer emotions more slowly than children without ADHD (d = 0.13), and their small magnitude impairment in accuracy (d = 0.30) was attributable to their generally less accurate performance on choice-response tasks (i.e., across both emotion and control conditions). Taken together, the evidence indicates that emotion inference abilities are likely unimpaired in pediatric ADHD and that working memory is implicated in the ability to infer emotion from context for all children-not just children with ADHD. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Representation of Emotion Inferences

ABSTRACT While research has repeatedly found evidence that readers infer characters’ emotions, we investigate three outstanding questions about the content and time course of such inferences. We ask whether even simple narratives give rise to emotion inferences, in what form such inferences are encoded into long-term memory, and whether they are uniquely bound to the character whose actions prompted the inference. To address these issues, we had participants read simple, sentence-long stories that allowed ready emotion inferences. Compared to performance on control stories, participants took longer to reject the implied emotion term in an immediate paradigm (Experiment 1) and were less accurate in a delayed paradigm when primed by the name of the character who experienced the emotion (Experiment 2). In Experiment 3, we primed participants in a delayed recognition paradigm with either the name of the character who experienced the emotion or a secondary character who did not. Participants were less accurate in rejecting the implied emotion term when primed by both character names. These results suggest that readers can encode emotion inferences based on simple narratives and that they encode those inferences into long-memory with minimal content. In addition, those emotion inferences may be activated from features of the general narrative situation, rather than only by the character whose actions or experiences prompted the inference.

Leveraging facial expressions and contextual information to investigate opaque representations of emotions.

Observers attribute emotions to others relying on multiple cues, including facial expressions and information about the situation. Recent research has used Bayesian models to study how these cues are integrated. Existing studies have used a variety of tasks to probe emotion inferences, but limited attention has been devoted to the possibility that different decision processes might be involved depending on the task. If this is the case, understanding emotion representations might require understanding the decision processes through which they give rise to judgments. This article 1) shows that the different tasks that have been used in the literature yield very different results, 2) proposes an account of the decision processes involved that explain the differences, and 3) tests novel predictions of this account. The results offer new insights into how emotions are represented, and more broadly demonstrate the importance of taking decision processes into account in Bayesian models of cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Inferential affective tracking reveals the remarkable speed of context-based emotion perception

Understanding the emotional states of others is important for social functioning. Recent studies show that context plays an essential role in emotion recognition. However, it remains unclear whether emotion inference from visual scene context is as efficient as emotion recognition from faces. Here, we measured the speed of context-based emotion perception, using Inferential Affective Tracking (IAT) with naturalistic and dynamic videos. Using cross-correlation analyses, we found that inferring affect based on visual context alone is just as fast as tracking affect with all available information including face and body. We further demonstrated that this approach has high precision and sensitivity to sub-second lags. Our results suggest that emotion recognition from dynamic contextual information might be automatic and immediate. Seemingly complex context-based emotion perception is far more efficient than previously assumed.

The Role of the Establishment of Causal Connections and the Modality of Presentation of Discourse in the Generation of Emotion Inferences by Argentine College Students

The purpose of this study was to examine the role of the causal connectivity of the statements (their total number of causal connections) and the modality of presentation of discourse (oral-written) in the generation of emotion inferences by Spanish-speaking students. With this aim, we asked a group of Argentine college students to either listen to or read an excerpt of a radio interview (on the topic ‘Argentine radio presenters’ relationship with the audience’), and to perform an emotions recall task. Results indicated that statements that promoted the generation of emotion inferences and had a high number of causal connections were more often included in the recall protocols than those that promoted the generation of these inferences and had a low number of them. They also indicated that participants who were presented with the spoken version of the materials included a higher number of statements in this task than those that were presented with the written version. These findings suggest that the establishment of meaningful discourse connections has a role in the generation of emotion inferences, and that there appear to be differences in the representation that listeners and readers construct of speakers’ emotions.

Youths’ processing of emotion information: Responses to chronic and video-based laboratory stress

Integrating multiple sources of information about others’ emotional states is critical to making accurate emotional inferences. There is evidence that both acute and chronic stress influence how individuals perceive emotional information. However, there is little research examining how acute and chronic stress interact to impact these processes. The current study examined whether acute and chronic stress interact to influence how children make emotional inferences. Eighty-nine youths (aged 11–15 years) underwent a novel video-based social stressor. Children completed an emotion recognition task prior to and after the stressor in which they saw integrated displays of facial expressions and contexts depicting congruent or incongruent emotional information. Eye tracking assessed changes in attention to the stimuli. Children became more likely to use and attended more to facial information than contextual information when labeling emotions following exposure to acute stress. Moreover, the effect of acute stress on use of facial information to label emotions was stronger for children who experienced higher levels of chronic stress. These data suggest that acute stress shifts attention towards facial information while suppressing processing of other sources of emotional information, and that youths with a history of chronic stress are more susceptible to these effects.

Influence of background facial expression on emotion inferences.

Influence of background facial expression on emotion inferences.

How Our Gaze Reacts to Another Person's Tears? Experimental Insights Into Eye Tracking Technology.

Crying is an ubiquitous human behavior through which an emotion is expressed on the face together with visible tears and constitutes a slippery riddle for researchers. To provide an answer to the question “How our gaze reacts to another person’s tears?,” we made use of eye tracking technology to study a series of visual stimuli. By presenting an illustrative example through an experimental setting specifically designed to study the “tearing effect,” the present work aims to offer methodological insight on how to use eye-tracking technology to study non-verbal cues. A sample of 30 healthy young women with normal visual acuity performed a within-subjects task in which they evaluated images of real faces with and without tears while their eye movements were tracked. Tears were found to be a magnet for visual attention in the task of facial attribution, facilitating a greater perception of emotional intensity. Moreover, the inspection pattern changed qualitatively and quantitatively, with our participants becoming fully focused on the tears when they were visible. The mere presence of a single tear running down a cheek was associated with an increased emotional inference and greater perception of sincerity. Using normalized and validated tools (Reading the Eyes in the Mind Test and the SALAMANCA screening test for personality disorders), we measured the influence of certain characteristics of the participants on their performance of the experimental task. On the one hand, a higher level of cognitive empathy helped to classify tearful faces with higher emotional intensity and tearless faces with less emotional intensity. On the other hand, we observed that less sincerity was attributed to the tearful faces as the SALAMANCA test scores rose in clusters A (strange and extravagant) and B (immature and emotionally unstable) of our sample. The present findings highlight the advantages of using eye tracking technology to study non-verbal cues and draw attention to methodological issues that should be taken into account. Further exploration of the relationship between empathy and tear perception could be a fruitful avenue of future research using eye tracking.

An Investigation of the Free Energy Principle for Emotion Recognition.

This paper offers a prospectus of what might be achievable in the development of emotional recognition devices. It provides a conceptual overview of the free energy principle; including Markov blankets, active inference, and—in particular—a discussion of selfhood and theory of mind, followed by a brief explanation of how these concepts can explain both neural and cultural models of emotional inference. The underlying hypothesis is that emotion recognition and inference devices will evolve from state-of-the-art deep learning models into active inference schemes that go beyond marketing applications and become adjunct to psychiatric practice. Specifically, this paper proposes that a second wave of emotion recognition devices will be equipped with an emotional lexicon (or the ability to epistemically search for one), allowing the device to resolve uncertainty about emotional states by actively eliciting responses from the user and learning from these responses. Following this, a third wave of emotional devices will converge upon the user's generative model, resulting in the machine and human engaging in a reciprocal, prosocial emotional interaction, i.e., sharing a generative model of emotional states.

Contingency in Teacher-Child Emotional State Talk during Shared Book Reading in Early Childhood Classrooms

ABSTRACT Research Findings: The aim of this study was to investigate emotional state talk between teachers and their students in 81 early childhood classrooms during shared book reading. Teachers shared with their students an experimental emotion narrative book. Preliminary analysis of teachers’ and children’s transcribed utterances during reading addressed the frequencies of emotional terms, form (question vs. comment), and content (emotion identification, emotion causality, emotion inference, or non-emotional content). Sequential analysis evaluated contingent relations between the book text and teachers’ comments and questions about emotions as well as between teachers’ and children’s contingent responses to each other. Results showed emotional excerpts from the book were significantly associated with teachers’ questions and comments about emotion identification. Children were highly specific and responsive to teachers’ emotional state talk, and children’s comments about emotions evoked further considerations from teachers about the same topics. Practice or Policy: Our results highlight the importance of the presence of emotional terms in the book text to stimulate emotional state talk. Given the value of emotional state talk for children’s emotional development, teachers must be informed of preschoolers’ ability to consistently participate in classroom discussions about emotions, even when complex topics like causality and inferences are addressed.

BioCNN: A Hardware Inference Engine for EEG-Based Emotion Detection

EEG-based emotion classifiers have the potential of significantly improving the social integration of patients suffering from neurological disorders such as Amyotrophic Lateral Sclerosis or the acute stages of Alzheimer's disease. Emotion classifiers have historically used software on general-purpose computers and operating under off-line conditions. Yet the wearability of such classifiers is a must if they are to enable the socialization of critical-care patients. Such wearability requires the use of low-power hardware accelerators that would enable near real-time classification and extended periods of operations. In this article, we architect, design, implement, and test a handcrafted, hardware Convolutional Neural Network, named BioCNN, optimized for EEG-based emotion detection and other bio-medical applications. The EEG signals are generated using a low-cost, off-the-shelf device, namely, Emotiv Epoc+, and then denoised and pre-processed ahead of their use by BioCNN. For training and testing, BioCNN uses three repositories of emotion classification datasets, including the publicly available DEAP and DREAMER datasets, along with an original dataset collected in-house from 5 healthy subjects using standard visual stimuli. A subject-specific training approach is used under TensorFlow to train BioCNN, which is implemented using the Digilent Atlys Board with a low-cost Spartan-6 FPGA. The experimental results show a competitive energy efficiency of 11 GOps/W, a throughput of 1.65 GOps that is in line with the real-time specification of a wearable device, and a latency of less than 1 ms, which is smaller than the 150 ms required for human interaction times. Its emotion inference accuracy is competitive with the top software-based emotion detectors.

Emotions in Argumentative Narration

This paper studies emotional inferencing triggered by emotion terms using Pragma-Dialectics and the Argumentum Model of Topics. The corpus, in French, is an excerpt of a video-recorded testimony in which a middle school teacher evokes her experience of being in class the day after the Charlie Hebdo attack, thus presenting a case of argumentation in context. The analysis focuses on the argumentative structure and on the rhetorical strategies that trigger emotional inferencing. The emotional inferencing derives from a Locus of Ontological Implication, which links a situation and an emotion (and vice-versa), while the culture-bound elements tend to be part of the endoxon.

Active inference, stressors, and psychological trauma: A neuroethological model of (mal)adaptive explore-exploit dynamics in ecological context

This paper offers a formal account of emotional inference and stress-related behaviour, using the notion of active inference. We formulate responses to stressful scenarios in terms of Bayesian belief-updating and subsequent policy selection; namely, planning as (active) inference. Using a minimal model of how creatures or subjects account for their sensations (and subsequent action), we deconstruct the sequences of belief updating and behaviour that underwrite stress-related responses – and simulate the aberrant responses of the sort seen in post-traumatic stress disorder (PTSD). Crucially, the model used for belief-updating generates predictions in multiple (exteroceptive, proprioceptive and interoceptive) modalities, to provide an integrated account of evidence accumulation and multimodal integration that has consequences for both motor and autonomic responses. The ensuing phenomenology speaks to many constructs in the ecological and clinical literature on stress, which we unpack with reference to simulated inference processes and accompanying neuronal responses. A key insight afforded by this formal approach rests on the trade-off between the epistemic affordance of certain cues (that resolve uncertainty about states of affairs in the environment) and the consequences of epistemic foraging (that may be in conflict with the instrumental or pragmatic value of ‘fleeing’ or ‘freezing’). Starting from first principles, we show how this trade-off is nuanced by prior (subpersonal) beliefs about the outcomes of behaviour – beliefs that, when held with unduly high precision, can lead to (Bayes optimal) responses that closely resemble PTSD.

Emotion Sensing for Mobile Computing

Emotion is a complex mental state that guides people's thoughts and behaviors. The ability to recognize one's own emotions and to understand others' emotions helps people manage their personal lives and social relations more successfully. Meanwhile, mobile and wearable devices have become indispensable tools for most of the population. They provide great opportunities for building emotional intelligent applications. In this article, we introduce how mobile and wearable devices work as sensors by leveraging their sensing, computing, and communication capabilities, which help monitor people's mental health, facilitate social interactions, and improve user experience. We first describe a general emotion sensing framework that consists of sensing, inferring, and responding steps. After reviewing the state-of-the-art systems on this topic, we summarize and discuss various sensing modalities enabled by mobile and wearable devices. We then describe the widely used inferring procedures and methods. Additionally, we present three solutions of facial expression recognition on smartphones as a case of emotion inference. Finally, we introduce emerging emotion sensing applications, and discuss challenges and opportunities.

Introducing ACASS: An Annotated Character Animation Stimulus Set for Controlled (e)Motion Perception Studies.

Others' movements inform us about their current activities as well as their intentions and emotions. Research on the distinct mechanisms underlying action recognition and emotion inferences has been limited due to a lack of suitable comparative stimulus material. Problematic confounds can derive from low-level physical features (e.g., luminance), as well as from higher-level psychological features (e.g., stimulus difficulty). Here we present a standardized stimulus dataset, which allows to address both action and emotion recognition with identical stimuli. The stimulus set consists of 792 computer animations with a neutral avatar based on full body motion capture protocols. Motion capture was performed on 22 human volunteers, instructed to perform six everyday activities (mopping, sweeping, painting with a roller, painting with a brush, wiping, sanding) in three different moods (angry, happy, sad). Five-second clips of each motion protocol were rendered into AVI-files using two virtual camera perspectives for each clip. In contrast to video stimuli, the computer animations allowed to standardize the physical appearance of the avatar and to control lighting and coloring conditions, thus reducing the stimulus variation to mere movement. To control for low level optical features of the stimuli, we developed and applied a set of MATLAB routines extracting basic physical features of the stimuli, including average background-foreground proportion and frame-by-frame pixel change dynamics. This information was used to identify outliers and to homogenize the stimuli across action and emotion categories. This led to a smaller stimulus subset (n = 83 animations within the 792 clip database) which only contained two different actions (mopping, sweeping) and two different moods (angry, happy). To further homogenize this stimulus subset with regard to psychological criteria we conducted an online observer study (N = 112 participants) to assess the recognition rates for actions and moods, which led to a final sub-selection of 32 clips (eight per category) within the database. The ACASS database and its subsets provide unique opportunities for research applications in social psychology, social neuroscience, and applied clinical studies on communication disorders. All 792 AVI-files, selected subsets, MATLAB code, annotations, and motion capture data (FBX-files) are available online.

Integrating expectations and outcomes: Preschoolers' developing ability to reason about others' emotions.

People's emotional experiences depend not only on what actually happened, but also on what they thought would happen. However, these expectations about future outcomes are not always communicated explicitly. Thus, the ability to infer others' expectations in context and understand how these expectations influence others' emotions is an important aspect of our social intelligence. Prior work suggests that an abstract understanding of how expectations modulate emotional responses may not emerge until 7 to 8 years of age. Using a novel paradigm that capitalizes on intuitive physics to generate contextually plausible expectations, we present evidence for expectation-based emotion inference in preschool-aged children. Given two bowlers who experienced identical final outcomes (hitting 3 of 6 pins), we varied the trajectory of their balls such that one would initially expect to hit all pins (high-expectation), while the other would expect to hit none (low-expectation). In Experiment 1, both 4- and 5-year-olds appropriately adjusted characters' happiness ratings upward (low-expectation) or downward (high-expectation) relative to their initial emotions; however, only 5-year-olds made adjustments robust enough to manifest as higher final ratings for the low-expectation than the high-expectation character. In Experiments 2-3, we replicate these results and show that 5-year-olds reliably differentiate the characters' emotions even when their expectations must be inferred from context. An internal meta-analysis revealed a robust and consistent effect across the three experiments. Together, these findings provide the earliest evidence for expectation-based emotion reasoning and suggest that the ability to spontaneously generate and consider others' expectations continues to develop during preschool years. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Inferring Emotions From Large-Scale Internet Voice Data

As voice dialog applications (VDAs, e.g., Siri, 1 1 http://www.apple.com/ios/siri/ . Cortana, 2 2 http://www.microsoft.com/en-us/mobile/campaign-cortana/ . Google Now 3 3 http://www.google.com/landing/now/ . ) are increasing in popularity, inferring emotions from the large-scale internet voice data generated from VDAs can help give a more reasonable and humane response. However, the tremendous amounts of users in large-scale internet voice data lead to a great diversity of users accents and expression patterns. Therefore, the traditional speech emotion recognition methods, which mainly target acted corpora, cannot effectively handle the massive and diverse amount of internet voice data. To address this issue, we carry out a series of observations, find suitable emotion categories for large-scale internet voice data, and verify the indicators of the social attributes (query time, query topic, and users location) and emotion inferring. Based on our observations, two different strategies are employed to solve the problem. First, a deep sparse neural network model that uses acoustic information, textual information, and three indicators (a temporal indicator, descriptive indicator, and geo-social indicator) as the input is proposed. Then, to capture the contextual information, we propose a hybrid emotion inference model that includes long short-term memory to capture the acoustic features and a latent dirichlet allocation to extract text features. Experiments on 93 000 utterances collected from the Sogou Voice Assistant 4 4 http://yy.sogou.com . (Chinese Siri) validate the effectiveness of the proposed methodologies. Furthermore, we compare the two methodologies and give their advantages and disadvantages.

The time course of emotion recognition in speech and music

The auditory gating paradigm was adopted to study how much acoustic information is needed to recognize emotions from speech prosody and music performances. In Study 1, brief utterances conveying ten emotions were segmented into temporally fine-grained gates and presented to listeners, whereas Study 2 instead used musically expressed emotions. Emotion recognition accuracy increased with increasing gate duration and generally stabilized after a certain duration, with different trajectories for different emotions. Above-chance accuracy was observed for ≤100 ms stimuli for anger, happiness, neutral, and sadness, and for ≤250 ms stimuli for most other emotions, for both speech and music. This suggests that emotion recognition is a fast process that allows discrimination of several emotions based on low-level physical characteristics. The emotion identification points, which reflect the amount of information required for stable recognition, were shortest for anger and happiness for both speech and music, but recognition took longer to stabilize for music vs speech. This, in turn, suggests that acoustic cues that develop over time also play a role for emotion inferences (especially for music). Finally, acoustic cue patterns were positively correlated between speech and music, suggesting a shared acoustic code for expressing emotions.