Emotion Inference Research Articles

U.S. Job-Seekers' Organizational Justice Perceptions of Emotion AI-Enabled Interviews

Emotion AI is increasingly used to automatically evaluate asynchronous hiring interviews. Although touted for increasing hiring fit and reducing bias, it is unclear how job-seekers perceive emotion AI-enabled asynchronous interviews. This gap is striking, given job-seekers' marginalized position in hiring and how job-seekers with marginalized identities may be particularly vulnerable to this technology's potential harms. Addressing this gap, we conducted exploratory interviews with 14 U.S.-based participants with direct, recent experience with emotion AI-enabled asynchronous interviews. While participants acknowledged the asynchronous, virtual modality's potential benefits to employers and job-seekers, they perceived harms to job-seekers associated with automatic emotion inferences that our analysis maps to distributive, procedural, and interactional injustices. We find that social identity can inform job-seekers' perceptions of emotion AI, extending prior work's understandings of the factors contributing to job-seekers' perceptions of AI (broadly) in hiring. Moreover, our results suggest that emotion AI use may reconfigure demands for emotional labor in hiring and that deploying this technology in its current state may unjustly risk harmful outcomes for job-seekers - or, at the very least, perceptions thereof, which shape behaviors and attitudes. Accordingly, we recommend against the present adoption of emotion AI in hiring, identifying opportunities for the design of future asynchronous hiring interview platforms to be meaningfully transparent, contestable, and privacy-preserving. We emphasize that only a subset of perceived harms we surface may be alleviated by these efforts; some injustices may only be resolved by removing emotion AI-enabled features.

Emotional Inference from Speech Signals Informed by Multiple Stream DNNs Based Non-Local Attention Mechanism

Emotional Inference from Speech Signals Informed by Multiple Stream DNNs Based Non-Local Attention Mechanism

Comparing Humans and Large Language Models on an Experimental Protocol Inventory for Theory of Mind Evaluation (EPITOME)

Abstract We address a growing debate about the extent to which large language models (LLMs) produce behavior consistent with Theory of Mind (ToM) in humans. We present EPITOME: a battery of six experiments that tap diverse ToM capacities, including belief attribution, emotional inference, and pragmatic reasoning. We elicit a performance baseline from human participants for each task. We use the dataset to ask whether distributional linguistic information learned by LLMs is sufficient to explain ToM in humans. We compare performance of five LLMs to a baseline of responses from human comprehenders. Results are mixed. LLMs display considerable sensitivity to mental states and match human performance in several tasks. Yet, they commit systematic errors in others, especially those requiring pragmatic reasoning on the basis of mental state information. Such uneven performance indicates that human-level ToM may require resources beyond distributional information.

Subthalamic Nucleus Stimulation Modulates Cognitive Theory of Mind in Parkinson's Disease.

Theory of mind (ToM), the ability to infer others' mental state, is essential for social interaction among human beings. It has been widely reported that both cognitive (inference of knowledge) and affective (inference of emotion) components of ToM are disrupted in Parkinson's disease (PD). Previous studies usually focused on the involvement of the prefrontal cortex. This study investigated the causal role of the subthalamic nucleus (STN), a key hub of the fronto-basal ganglia loops, in ToM. Thirty-four patients with idiopathic PD (15 women, aged 62.2 ± 8.3 years) completed a Yoni task with deep brain stimulation (DBS) ON and OFF. The Yoni task was designed to separate the cognitive and affective components of ToM. Volumes of tissue activated (VTA) were computed for three subregions of the STN. DBS showed insignificant effects on ToM inference costs at the group level, which may be due to the large interindividual variability. The associative VTA correlated with the cognitive inference cost change but not the affective inference cost change. Patients with greater associative STN stimulation infer more slowly on cognitive ToM. Stimulating associative STN can adversely affect cognitive ToM in PD patients, especially in patients with a wide range of stimulation (≥0.157) or cognitive decline (Montreal Cognitive Assessment < 26). The associative STN plays a causal role in cognitive ToM in patients with PD. However, stimulating the associative STN likely impairs cognitive ToM and potentially leads to social interaction deficits in PD. © 2024 International Parkinson and Movement Disorder Society.

Face and context integration in emotion inference is limited and variable across categories and individuals

The ability to make nuanced inferences about other people’s emotional states is central to social functioning. While emotion inferences can be sensitive to both facial movements and the situational context that they occur in, relatively little is understood about when these two sources of information are integrated across emotion categories and individuals. In a series of studies, we use one archival and five empirical datasets to demonstrate that people could be integrating, but that emotion inferences are just as well (and sometimes better) captured by knowledge of the situation alone, while isolated facial cues are insufficient. Further, people integrate facial cues more for categories for which they most frequently encounter facial expressions in everyday life (e.g., happiness). People are also moderately stable over time in their reliance on situational cues and integration of cues and those who reliably utilize situation cues more also have better situated emotion knowledge. These findings underscore the importance of studying variability in reliance on and integration of cues.

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Emotion recognition in conversations (ERC) has emerged as an important research area in Natural Language Processing and Affective Computing, focusing on accurately identifying emotions within the conversational utterance. Conventional approaches typically rely on labeled training samples for fine-tuning pre-trained language models (PLMs) to enhance classification performance. However, the limited availability of labeled data in real-world scenarios poses a significant challenge, potentially resulting in diminished model performance. In response to this challenge, we present the Multi-modal Attentive Prompt (MAP) learning framework, tailored specifically for few-shot emotion recognition in conversations. The MAP framework consists of four integral modules: multi-modal feature extraction for the sequential embedding of text, visual, and acoustic inputs; a multi-modal prompt generation module that creates six manually-designed multi-modal prompts; an attention mechanism for prompt aggregation; and an emotion inference module for emotion prediction. To evaluate our proposed model’s efficacy, we conducted extensive experiments on two widely recognized benchmark datasets, MELD and IEMOCAP. Our results demonstrate that the MAP framework outperforms state-of-the-art ERC models, yielding notable improvements of 3.5% and 0.4% in micro F1 scores. These findings highlight the MAP learning framework’s ability to effectively address the challenge of limited labeled data in emotion recognition, offering a promising strategy for improving ERC model performance.

Zoti's Social Toolkit: Developing and piloting novel animated tasks to assess emotional understanding and conflict resolution skills in childhood.

Current methods used to investigate emotional inference and conflict resolution knowledge are limited in their suitability for use with children with language disorders due to a reliance on language processing. This is problematic, as nearly 8% of the population are estimated to have developmental language disorder (DLD). In this paper, we present 'Zoti's Social Toolkit', a set of animated scenarios that can be used to assess emotion inferencing and conflict resolution knowledge. All animated scenarios contain interpersonal situations centred around a gender-neutral alien named Zoti. Four studies investigated the face and construct validity of the stimuli. The final stimulus set can be used with children, who may or may not have language difficulties and is openly available for use in research.

Broad-deep network-based fuzzy emotional inference model with personal information for intention understanding in human–robot interaction

A broad-deep fusion network-based fuzzy emotional inference model with personal information (BDFEI) is proposed for emotional intention understanding in human–robot interaction. It aims to understand students’ intentions in the university teaching scene. Initially, we employ convolution and maximum pooling for feature extraction. Subsequently, we apply the ridge regression algorithm for emotional behavior recognition, which effectively mitigates the impact of complex network structures and slow network updates often associated with deep learning. Moreover, we utilize multivariate analysis of variance to identify the key personal information factors influencing intentions and calculate their influence coefficients. Finally, a fuzzy inference method is employed to gain a comprehensive understanding of intentions. Our experimental results demonstrate the effectiveness of the BDFEI model. When compared to existing models, namely FDNNSA, ResNet-101+GFK, and HCFS, the BDFEI model achieved superior accuracy on the FABO database, surpassing them by 12.21%, 1.89%, and 0.78%, respectively. Furthermore, our self-built database experiments yielded an impressive 82.00% accuracy in intention understanding, confirming the efficacy of our emotional intention inference model.

Preschoolers’ ability to build inferences about a film protagonist’s emotional state

In this study, we investigated whether preschool children can draw inferences about the emotional state of a film protagonist. A sample of 59 children aged 4 and 5 watched 20 scenes from a television show. Their ability to understand the protagonist’s emotional states was assessed using a reaction time task. Media sign literacy, theory of mind abilities, and the general understanding of the scenes were also evaluated. Our results show that even children aged 4 drew emotional inferences but only on a valence-based level (i.e., distinguishing between emotions of different valences). No exact inferences (i.e., distinguishing between emotions of the same valence) were drawn. There were no significant influences of media sign literacy or theory of mind on the inference generation, but general understanding of the film scenes was important.

Emotions on Social Media: A Sentiment Analysis Approach Based on Twitter (X) Data on the Russian-Ukraine War

Twitter (X) is an important tool that reflects the feelings and attitudes of the public. For this reason, in this study, especially when it comes to events that concern society, Twitter provides an opportunity to both follow the agenda and understand the reactions through instant sharing. Twitter is a social media platform that allows the public to convey their feelings, thoughts, and attitudes to the masses. Twitter provides the opportunity to stay up-to-date and understand reactions through instant posts, especially for social events. In this research, Twitter posts made with the Ukraine hashtag between March 1 and April 30, 2022, during the Russia-Ukraine War, were eliminated with the "war" filter, and the expressions were analyzed using the sentiment analysis method. Various URLs were eliminated, and research was carried out on ten thousand tweets. The tweets obtained were categorized as positive, negative, and neutral. Accordingly, the expressions containing positive, negative, and neutral emotions were analyzed by determining the emotional inferences of the words in the tweets with an artificial intelligence algorithm and then detailed by the researchers with content analysis. In this sense, this study becomes important in understanding how the masses express their reactions through emotional social media platforms and what their emotions are in this process. Therefore, this research can be a clue for the consequences of international war on the masses.

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models.

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

Emotional event perception is related to lexical complexity and emotion knowledge

Inferring emotion is a critical skill that supports social functioning. Emotion inferences are typically studied in simplistic paradigms by asking people to categorize isolated and static cues like frowning faces. Yet emotions are complex events that unfold over time. Here, across three samples (Study 1 N = 222; Study 2 N = 261; Study 3 N = 101), we present the Emotion Segmentation Paradigm to examine inferences about complex emotional events by extending cognitive paradigms examining event perception. Participants were asked to indicate when there were changes in the emotions of target individuals within continuous streams of activity in narrative film (Study 1) and documentary clips (Study 2, preregistered, and Study 3 test-retest sample). This Emotion Segmentation Paradigm revealed robust and reliable individual differences across multiple metrics. We also tested the constructionist prediction that emotion labels constrain emotion inference, which is traditionally studied by introducing emotion labels. We demonstrate that individual differences in active emotion vocabulary (i.e., readily accessible emotion words) correlate with emotion segmentation performance.

Social cognition and emotion inference abilities in healthy aging and Mild Cognitive Impairment

AbstractBackgroundSocial cognition (SC) is a neurocognitive domain that can be affected by the pathophysiological changes of aging, and deteriorate in individuals diagnosed with Mild Cognitive Impairment (MCI) and Alzheimer’s Disease. This social‐cognitive function involves the attribution of one’s mental states, including intentions (IA) and emotions (EA). As mentalization abilities may impact on daily functioning, research is focusing on SC processes in individuals at risk for age‐related neurocognitive decline. The present study aimed to 1) investigate SC abilities in MCI, compared to healthy individuals; 2) explore whether specific SC sub‐domains could be associated with MCI diagnosis.MethodN = 468 outpatients referred to Bari University Hospital for neuropsychological assessment were classified into two groups: i) a cognitively spared (n = 166; Mage = 61); ii) MCI (n = 302; Mage = 67), characterized by objective deficits one or more neurocognitive domains (other than SC) not interfering with daily functioning. Sample classification resulted from multidomain, cognitive‐functional assessment procedures consistent with Petersen’s criteria (Petersen et al., 2014). A total of 185 subjects also completed the Story‐Based Empathy Task (SET), yielding four social cognition scores (i.e., global score [GS]; IA; EA; causal inference [CI]). After testing the distribution of the SET variable through the Shapiro‐Wilk test, the Mann‐Whitney U test was used to analyze mean differences between the two groups. Additionally, a binary logistic regression was conducted to assess whether impaired SC could be associated with MCI diagnosis.ResultPatients with MCI showed significantly lower SC performances in all SET sub‐tests, compared to controls (SET‐GS: Z = ‐5.25; p&lt;.001; SET‐IA: Z = ‐3.98; p&lt;.001; SET‐EA: Z = ‐4.02; p&lt;.001; SET‐CI: Z = ‐3.22; p&lt;.001). Binary logistic regression with backward elimination revealed that among all SC sub‐domains, the inability to infer emotional states resulted being associated with MCI diagnosis (B = ‐.678; exp(B) = .507; p&lt;.001). Regression analysis was overall significant (χ2 = 16.68; p&lt;.001), and correctly classified 66% of cases.ConclusionOur findings highlighted significantly poorer SC performances in MCI, notably in the emotional inference sub‐domain. This preliminary evidence may contribute to the clinical understanding of social‐cognitive functioning in elderly individuals with mild/initial cognitive deficits, emphasizing the importance of accounting for SC in neuropsychological assessment to improve early diagnosis and strategies for dementia prevention.

서사문의 추론 단서 유무에 따른 인물 감정 추론의 차이

Purpose: This study aimed to analyze the differences in personal emotion inference based on the presence of inference cues, confirming the possibility and validity of textbook revisions to enhance reading comprehension.&#x0D; Methods: A test consisting of passages without inference cues (Type X) was administered to 497 high school students, while a test consisting of passages with inference cues (Type O) was administered to 477 high school students to assess personal emotion inference. Based on the response data, differencesin personal emotion inference were examined by test type, gender, proficiency level, and emotion type.&#x0D; Results: Firstly, students who read passages with inference cues demonstrated more accurate personal emotion inference compared to those who read passages without inference cues. Secondly, the average differences between passages with and without inference cues were confirmed across gender andproficiency levels. Thirdly, inference cues were more effective in facilitating negative emotion inference compared to positive emotions.&#x0D; Conclusion: The addition of inference cues to narrative texts should be considered as an innovative approach for textbook revisions aimed at enhancing reading comprehension.

A broad-deep fusion network-based fuzzy emotional intention inference model for teaching validity evaluation

A broad-deep fusion network-based fuzzy emotional inference model with personal information (BDFEI) is proposed for emotional intention understanding in human-robot interaction. It aims to understand students' intentions and evaluate the teaching validity through quantitative analysis in the university teaching scene. Specifically, broad-deep fusion network is carried out for facial emotion and behavior recognition. Then, fuzzy inference with personal information is used to understand the intention. The V-A emotion space is used to quantify students' studying status with originality, by mapping the output of Softmax. Finally, teaching validity is obtained by fuzzy inference. According to the recognition results, the accuracy on bimodal face and body gesture database (FABO) of our proposal is 12.21%, 1.89% and 0.78% higher than those of the sparse autoencoder-based fuzzy deep neural network (FDNNSA), geodesic flow kernel integrated with residual network (ResNet-101+GFK), the hierarchical classification fusion strategy (HCFS), respectively. Through the experiment on the self-built database, the accuracy of intention understanding is 90.48%, and the effectiveness of teaching validity evaluation method is confirmed.

The Role of the Establishment of Causal Connections and Elaboration Question Answering Tasks in the Comprehension of Spontaneous Narrative Discourse by Argentine College Students

The purpose of this study was to examine the role of the causal connectivity of the statements (low-medium-high), elaboration question condition (focused on the identification of main ideas -focused on the identification of speakers’ emotions) and the modality of presentation of discourse (oral-written) in the comprehension of spontaneous narrative discourse by Spanish-speaking students. With this aim, Argentine college students were asked to listen or read a radio interview, and to answer 4 elaboration questions. These questions were presented after statements that had high causal connectivity and promoted the generation of emotion inferences. Then, they were asked to perform a free-recall and a comprehender’s emotions elaboration task. Statements that had high causal connectivity were more often included in the free recall task than those that had low causal connectivity. And, students that elaborated on main ideas included a higher number of statements than those that elaborated on speakers’ emotions. In the comprehenders’ emotions elaboration task, students who elaborated on speakers’ emotions and read the material included a higher number of statements than those that listened to it. And, the effect of causal connectivity was greater when students read than when they listened to the material.

Survey on Emotion Sensing Using Mobile Devices

The rapid development and ubiquity of mobile and wearable devices promises to enable researchers to monitor users' granular emotional data in a less intrusive manner. Researchers have used a wide variety of mobile and wearable devices for this purpose, and have proposed various approaches to sense users' emotional states. In this survey, we utilise three established digital libraries ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ACM Digital Library</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IEEE Xplore Digital Library</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Springer Nature</i> ). We analysed and critically assessed the different approaches used in the three stages (perception, learning, inference) of a typical mobile emotion sensing framework, following a structured paper selection process. The contribution of this survey is three-fold; first, we document all the latest relevant literature on mobile emotion sensing research; second, we describe how mobile and wearable devices use their sensing and computing capabilities to monitor human emotions; third, we discuss challenges and opportunities of mobile emotion sensing to demonstrate the potential of this thriving field of research.

Autonomic Regulation of Facial Temperature during Stress: A Cross-Mapping Analysis.

Skin temperature reflects the Autonomic Nervous System (ANS)'s response to emotions and mental states and can be remotely measured using InfraRed Thermography. Understanding the physiological mechanisms that affect facial temperature is essential to improve the precision of emotional inference from thermal imaging. To achieve this aim, we recorded thermal images from 30 volunteers, at rest and under acute stress induced by the Stroop test, together with two autonomic correlates, i.e., heart rate variability and electrodermal activity, the former serving as a measure of cardiovascular dynamics, and the latter of the activity of the sweat glands. We used a Cross Mapping (CM) approach to quantify the nonlinear coupling of the temperature from four facial regions with the ANS correlates. CM reveals that facial temperature has a statistically significant correlation with the two autonomic time series, under both conditions, which was not evident in the linear domain. In particular, compared to the other regions, the nose shows a significantly higher link to the electrodermal activity in both conditions, and to the heart rate variability under stress. Moreover, the cardiovascular activity seems to be primarily responsible for the well-known decrease in nose temperature, and its coupling with the thermal signals significantly varies with gender.

EC-GAN: Emotion-Controllable GAN for Face Image Completion

Image completion methods based on deep learning, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), have succeeded in producing semantically plausible results. However, existing facial image completion methods can either produce only one result or, although they can provide multiple results, cannot attribute particular emotions to the results. We propose EC-GAN, a novel facial Emotion-Controllable GAN-based image completion model that can infer and customize generative facial emotions. We propose an emotion inference module that infers the emotions of faces based on the unmasked regions of the faces. The emotion inference module is trained in a supervised manner and enforces the encoder to disentangle the emotion semantics from the native latent space. We also developed an emotion control module to modify the latent codes of emotions, moving the latent codes of the initial emotion toward the desired one while maintaining the remaining facial features. Extensive experiments were conducted on two facial datasets, CelebA-HQ and CFEED. Quantitative and qualitative results indicate that EC-GAN produces images with diverse desired expressions even when the main features of the faces are masked. On the other hand, EC-GAN promotes semantic inference capability with irregularly masked holes, resulting in more natural facial expressions.

A cognitive emotion model enhanced sequential method for social emotion cause identification

Social emotion refers to the emotion evoked to the reader by a textual document. In contrast to the emotion cause extraction task which analyzes the cause of the author's sentiments based on the expressions in text, identifying the causes of social emotion evoked to the reader from text has not been explored previously. Social emotion mining and its cause analysis is not only an important research topic in Web-based social media analytics and text mining but also has a number of applications in multiple domains. As the focus of social emotion cause identification is on analyzing the causes of the reader's emotions elicited by a text that are not explicitly or implicitly expressed, it is a challenging task fundamentally different from the previous research. To tackle this, it also needs a deeper level understanding of the cognitive process underlying the inference of social emotion and its cause analysis. In this paper, we propose the new task of social emotion cause identification (SECI). Inspired by the cognitive structure of emotions (OCC) theory, we present a Cognitive Emotion model Enhanced Sequential (CogEES) method for SECI. Specifically, based on the implications of the OCC model, our method first establishes the correspondence between words/phrases in text and emotional dimensions identified in OCC and builds the emotional dimension lexicons with 1,676 distinct words/phrases. Then, our method utilizes lexicons information and discourse coherence for the semantic segmentation of document and the enhancement of clause representation learning. Finally, our method combines text segmentation and clause representation into a sequential model for cause clause prediction. We construct the SECI dataset for this new task and conduct experiments to evaluate CogEES. Our method outperforms the baselines and achieves over 10% F1 improvement on average, with better interpretability of the prediction results.