Emotion Labels Research Articles

Adolescents and young adults differ in their neural response to and recognition of adolescent and adult emotional faces

Peer relationships become increasingly important during adolescence. The success of these relationships may rely on the ability to attend to and decode subtle or ambiguous emotional expressions that are common in social interactions. However, most studies examining youths' processing and labeling of facial emotion have employed adult faces and faces that depict emotional extremes as stimuli. In this study, 40 adolescents and 40 young adults viewed blends of angry-neutral, fearful-neutral, and happy-neutral faces (e.g., 100% angry, 66% angry, 33% angry, neutral) portrayed by adolescent and adult actors as electroencephalogram (EEG) was recorded. Participants also labeled these faces according to the emotion expressed (i.e., angry, fearful, happy, or neutral). The Late Positive Potential (LPP), an event-related potential (ERP) component that reflects sustained attention to motivationally salient information, was scored from the EEG following face presentation. Among adolescents, as peer-age faces moved from ambiguous (33%) to unambiguous (100%) emotional expression, the LPP similarly increased. These effects were not found when adolescents viewed emotional face blends portrayed by adult actors. Additionally, while both adolescents and young adults showed greater emotion labeling accuracy as faces increased in emotional intensity from ambiguous to unambiguous emotional expression, adolescent participants did not show greater accuracy when labeling peer-compared to adult-age faces. Together, these data suggest that adolescents attend more to subtle differences in peer-age emotional faces, but they do not label these emotional expressions more accurately than adults.

Changes in functional connectivity associated with facial expression processing over the working adult lifespan

The recognition of negative emotions from facial expressions is shown to decline across the adult lifespan, with some evidence that this decline begins around middle age. While some studies have suggested ageing may be associated with changes in neural response to emotional expressions, it is not known whether ageing is associated with changes in the network connectivity associated with processing emotional expressions. In this study, we examined the effect of participant age on whole-brain connectivity to various brain regions that show connectivity during emotion processing, namely, the left and right amygdalae, medial prefrontal cortex (mPFC), and right posterior superior temporal sulcus (rpSTS). The study involved healthy participants aged 20–65 years who completed an implicit affect processing task involving facial expressions displaying anger, fear, happiness, and neutral expressions during functional magnetic resonance imaging (fMRI). Participants were also tested on recognition accuracy during an emotion labelling task. We found that participant age was negatively associated with connectivity between the left amygdala and voxels in the left occipital pole; between the rpSTS and voxels in the orbitofrontal cortex; and between the mPFC and cingulate cortex and left insular cortex. Furthermore, these effects were due to a greater age-related decline in brain connectivity for negative expressions compared to happy and neutral expressions. There was, however, no significant relationship between age and emotion recognition accuracy (though in the expected direction), and between connectivity strength and emotion recognition accuracy. Together, these results provide evidence for a specific age-related decline in the neural processing of negative emotions, and could suggest that changes in underlying network connectivity over the working adult lifespan might occur before any marked decline in the recognition of the emotion.

Toddlers‘ categorization of emotional facial expressions by emotion category and approach-avoidance orientation

ABSTRACT Objective It has been observed that toddlers tend to miscategorize several negative emotions (e.g., sadness, anger). On the other hand, when we infer others’ emotions, an approach-avoidance tendency toward facial expressions is perceived antecedently before linguistically categorizing them into emotion-word-based categories, such as sadness, anger, and fear. Therefore, children may have categorized facial expressions in an approach-avoidance framework before they were able to classify them in emotion-word-based categories. Method The present study explored this possibility by asking 3-year-old Japanese children (N = 36) to categorize facial expressions using both an emotion label and action response (i.e., approach-avoidance) in the same procedure. Results Results revealed that while toddlers were confused by negative facial expressions using emotion labels, they could distinguish them using action responses like adults. Furthermore, children’s categorization of facial expression by emotion label was partially similar to categorization by approach-avoidance framework. Conclusions These findings suggest that although toddlers have difficulties in distinguishing between several negative facial expressions by emotion-word-category, they can distinguish how to react to these facial expressions just like an adult. Furthermore, toddlers might construct emotion-word-based categories based on approach-avoidance in early childhood. Finally, the developmental trajectories of categorization of facial expression by emotion-word-based category and approach-avoidance response are discussed. KEY POINTS What is already known about this topic: The skill of reading emotions from facial expressions develops in early childhood. In early childhood, some negative facial expressions are confused with each other. Early young children might recognize facial expressions based on the axis of approaching or avoiding. What this topic adds: This study examines how to classify facial expressions in 3-year-old children. While 3-year-olds confused negative emotions such as anger and fear, they differentiated between facial expressions on the axis of approaching or avoiding the facial expression. Categorizations based on emotional categories and approach-avoidance are similar, suggesting that children may categorize facial expressions based on the approach-avoidance axis in early childhood.

An Intelligent Vehicle Alarm User Terminal System Based on Emotional Identification Technology

Negative emotions could increase the risks of traffic accidents. However, the driver’s emotional identification is rarely considered in the current design of intelligent vehicle alarm user terminals (IVAUTs). To solve the problem, this paper tries to design an IVAUT system based on emotional identification technology. Firstly, the transformer network was combined with a convolutional neural network (CNN) into a voice emotional identification system for intelligent vehicle alarm, and an emotional labeling approach was provided. Next, a bimodal fusion model was developed based on decompose-CNNs, which includes an appearance module, an optical flow module, and a bimodal fusion module. The proposed emotional identification approach was proved effective through experiments.

Learning to generate emotional music correlated with music structure features

Music can be regarded as an art of expressing inner feelings. However, most of the existing networks for music generation ignore the analysis of its emotional expression. In this paper, we propose to synthesise music according to the specified emotion, and also integrate the internal structural characteristics of music into the generation process. Specifically, we embed the emotional labels along with music structure features as the conditional input and then investigate the GRU network for generating emotional music. In addition to the generator, we also design a novel perceptually optimised emotion classification model which aims for promoting the generated music close to the emotion expression of real music. In order to validate the effectiveness of the proposed framework, both the subjective and objective experiments are conducted to verify that our method can produce emotional music correlated to the specified emotion and music structures.

Emotion words link faces to emotional scenarios in early childhood.

Recent theories have suggested that emotion words may facilitate the development of emotion concepts. The present study investigates whether emotion words affect children's performance on an emotion category learning task. Across two experiments, 72 three-year-old children (49 female) were asked to identify which emotional face best matched particular emotional scenarios during nine pretest and nine posttest trials. The scenarios in the present studies aligned with emotions typically learned among older age groups (annoyed, disgusted, and nervous). Between pretest and posttest, children participated in training in which a facial configuration (annoyed, disgusted, or nervous) was paired with an associated scenario while they heard the emotion labeled explicitly or heard irrelevant information (Experiment 1) or heard a broad emotion label versus irrelevant information (Experiment 2). Aside from the labels presented, all other information was kept the same across conditions, including the specific faces and scenarios heard during learning trials. In Experiment 1, children's emotion understanding increased more from pretest to posttest in the explicit label versus irrelevant condition, t(34) = 2.26, p = .030, d = .75, but in Experiment 2 the broad emotion labels did not provide an advantage over irrelevant information, t(34) = .72, p = .474, d = .24. These results suggest that emotion labels may be particularly helpful for young children learning about unfamiliar emotions, because specific labels may help children to aggregate disparate emotional information into meaningful categories. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Investigating EEG-based functional connectivity patterns for multimodal emotion recognition

Objective. Previous studies on emotion recognition from electroencephalography (EEG) mainly rely on single-channel-based feature extraction methods, which ignore the functional connectivity between brain regions. Hence, in this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. Approach. After constructing the brain networks by the correlations between pairs of EEG signals, we calculated critical subnetworks through the average of brain network matrices with the same emotion label to eliminate the weak associations. Then, three network features were conveyed to a multimodal emotion recognition model using deep canonical correlation analysis along with eye movement features. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public datasets: SEED, SEED-V, and DEAP. Main results. The experimental results reveal that the strength feature outperforms the state-of-the-art features based on single-channel analysis. The classification accuracies of multimodal emotion recognition are on the SEED dataset, on the SEED-V dataset, and and for arousal and valence on the DEAP dataset, respectively, which all achieved the best performance. In addition, the brain networks constructed with 18 channels achieve comparable performance with that of the 62-channel network and enable easier setups in real scenarios. Significance. The EEG functional connectivity networks combined with emotion-relevant critical subnetworks selection algorithm we proposed is a successful exploration to excavate the information between channels.

A Multi-Scale Multi-Task Learning Model for Continuous Dimensional Emotion Recognition from Audio

Due to the advantages of many aspects of the dimensional emotion model, continuous dimensional emotion recognition from audio has attracted increasing attention in recent years. Features and dimensional emotion labels on different time scales have different characteristics and contain different information. To make full use of the advantages of features and emotion representations from multiple time scales, a novel multi-scale multi-task (MSMT) learning model is proposed in this paper. The MSMT model is constructed by a deep belief network (DBN) with only one hidden layer. The same hidden layer parameters and linear layer parameters are shared by all features. Multiple temporal pooling operations are inserted between the hidden layer and the linear layer to obtain information on multiple time scales. The mean squared error (MSE) of the main and the secondary task are combined to form the final objective function. Extensive experiments were conducted on RECOLA and SEMAINE datasets to illustrate the effectiveness of our model. The results for the two sets show that even adding a secondary scale to the scale with optimal single-scale single-task performance can achieve significant performance improvements.

Facial-expression-aware Emotional Color Transfer Based on Convolutional Neural Network

Emotional color transfer aims to change the evoked emotion of a source image to that of a target image by adjusting color distribution. Most of existing emotional color transfer methods only consider the low-level visual features of an image and ignore the facial expression features when the image contains a human face, which would cause incorrect emotion evaluation for the given image. In addition, previous emotional color transfer methods may easily result in ambiguity between the emotion of resulting image and target image. For example, if the background of the target image is dark while the facial expression is happiness, then previous methods would directly transfer dark color to the source image, neglecting the facial emotion in the image. To solve this problem, we propose a new facial-expression-aware emotional color transfer framework. Given a target image with facial expression features, we first predict the facial emotion label of the image through the emotion classification network. Then, facial emotion labels are matched with pre-trained emotional color transfer models. Finally, we use the matched emotion model to transfer the color of the target image to the source image. Considering none of the existing emotion image databases, which focus on images that contain face and background, we built an emotion database for our new emotional color transfer framework that is called “Face-Emotion database.” Experiments demonstrate that our method can successfully capture and transfer facial emotions, outperforming state-of-the-art methods.

A Comprehensive Investigation on Emotional Detection in Deep Learning

Emotion recognition is a substantial problem in the field of Big Data. In a wide range of applications, reliable categorization, analysis, and interpretation of emotional content is greatly desired. In this paper, we look at how Deep Learning models perform on an emotion perception test. Facial Expression Recognition (FER) plays an important role in machine learning tasks. Deep Learning models do well in FER tasks, but they lack explanation for their conclusions. Based on the notion that facial expression is a mixture of facial muscle movements, we discover a link between Facial Action Coding Units (AUs) and Emotion label in the CK+ Dataset. In this study, we offer a model that uses AUs to explain the classification outcomes of a Convolutional Neural Network (CNN) model. The CNN model is trained using the CK+ Dataset and identifies emotions using extracted characteristics. The CNN model's retrieved features and emotion classes are used by the explanation model to classify multiple AUs. Explanation model creates AUs quite effectively with only characteristics and emotion classes acquired from the CNN model, according to our trial. Experimental research was constructed, and several deep learning approaches were tested using publically available datasets. The findings are highly intriguing, highlighting the benefits of each strategy and training style studied.

Investigating the impact of emotion on temporal orientation in a deep multitask setting

Temporal orientation is an important aspect of human cognition which shows how an individual emphasizes past, present, and future. Theoretical research in psychology shows that one’s emotional state can influence his/her temporal orientation. We hypothesize that measuring human temporal orientation can benefit from concurrent learning of emotion. To test this hypothesis, we propose a deep learning-based multi-task framework where we concurrently learn a unified model for temporal orientation (our primary task) and emotion analysis (secondary task) using tweets. Our multi-task framework takes users’ tweets as input and produces three temporal orientation labels (past, present or future) and four emotion labels (joy, sadness, anger, or fear) with intensity values as outputs. The classified tweets are then grouped for each user to obtain the user-level temporal orientation and emotion. Finally, we investigate the associations between the users’ temporal orientation and their emotional state. Our analysis reveals that joy and anger are correlated to future orientation while sadness and fear are correlated to the past orientation.

Discovering emotion and reasoning its flip in multi-party conversations using masked memory network and transformer

Efficient discovery of a speaker’s emotional states in a multi-party conversation is significant to design human-like conversational agents. During a conversation, the cognitive state of a speaker often alters due to certain past utterances, which may lead to a flip in their emotional state. Therefore, discovering the reasons (triggers) behind the speaker’s emotion-flip during a conversation is essential to explain the emotion labels of individual utterances. In this paper, along with addressing the task of emotion recognition in conversations (ERC), we introduce a novel task – Emotion-Flip Reasoning (EFR), that aims to identify past utterances which have triggered one’s emotional state to flip at a certain time. We propose a masked memory network to address the former and a Transformer-based network for the latter task. To this end, we consider MELD, a benchmark emotion recognition dataset in multi-party conversations for the task of ERC, and augment it with new ground-truth labels for EFR. An extensive comparison with five state-of-the-art models suggests improved performances of our models for both the tasks. We further present anecdotal evidence and both qualitative and quantitative error analyses to support the superiority of our models compared to the baselines.

Assessing the Power of Words to Facilitate Emotion Category Learning.

The online version contains supplementary material available at 10.1007/s42761-021-00084-4.

Enhancing Segment-Based Speech Emotion Recognition by Iterative Self-Learning

Despite the widespread utilization of deep neural networks (DNNs) for speech emotion recognition (SER), they are severely restricted due to the paucity of labeled data for training. Recently, segment-based approaches for SER have been evolving, which train backbone networks on shorter segments instead of whole utterances, and thus naturally augments training examples without additional resources. However, one core challenge remains for segment-based approaches: most emotional corpora do not provide ground-truth labels at the segment level. To supervisely train a segment-based emotion model on such datasets, the most common way assigns each segment the corresponding utterance’s emotion label. However, this practice typically introduces noisy (incorrect) labels as emotional information is not uniformly distributed across the whole utterance. On the other hand, DNNs have been shown to easily over-fit a dataset when being trained with noisy labels. To this end, this work proposes a simple and effective iterative self-learning (ISL) framework, which comprises a procedure to progressively correct segment-level labels in an iterative learning manner. The ISL method produces dynamically-generated and soft emotion labels, leading to significant performance improvements. Experiments on three well-known emotional corpora demonstrate noticeable gains using the proposed method.

Symbolic Music Generation Conditioned on Continuous-Valued Emotions

In this paper we present a new approach for the generation of multi-instrument symbolic music driven by musical emotion. The principal novelty of our approach centres on conditioning a state-of-the-art transformer based on continuous-valued valence and arousal labels. In addition, we provide a new large-scale dataset of symbolic music paired with emotion labels in terms of valence and arousal. We evaluate our approach in a quantitative manner in two ways, first by measuring its note prediction accuracy, and second via a regression task in the valence-arousal plane. Our results demonstrate that our proposed approaches outperform conditioning using control tokens which is representative of the current state of the art.

A Study of High and Low Behavior Inhibitory Control on Emotional Labeling Effect from ERPs

Emotional labeling effect refers to the reduction of Emotion in labeling or naming emotional features of stimulus. Previous researches have explored whether the ability of behavioral inhibition affects the regulation of explicit emotions, while there are no studies learning about the impact of behavioral inhibition on implicit emotions regulation. This study used ERPs to examine whether individuals with high and low behavioral inhibition control ability differed in their emotional labeling effects. Dual-choice Oddball task was used to screen out groups with high and low behavioral inhibition ability, and they were asked to make judgments on faces with different emotions and genders, and the LPP amplitude after picture presentation was analyzed. The results showed that the LPP amplitude induced by high inhibition ability group was significantly lower than that induced by gender labeling, while the LPP amplitude induced by low inhibition ability group was not significantly different between emotion labeling and gender labeling. The results show that individuals with high inhibition ability can produce emotion labeling effect, and the higher the inhibition ability of individuals, the better the emotional labeling effect.

Disentangling Emotions during the Coronavirus Outbreak in Spain: Inner Emotions, Descriptive Feeling Rules and Socioemotional Conventions.

For constructionism, language is the link among different levels of analysis of emotional events, from individual to interpersonal and macrosocial. The interaction among these emotional levels allows us to construe an emotional episode and label it with an emotion word, coordinate with the emotions perceived in others, and represent events as a society. Across two studies, we found similarities and differences among inner emotions experienced (individual level), emotions perceived in others (descriptive feeling rules, interpersonal level) and emotions shared on the internet (socioemotional conventions, macrosocial level), with all these emotional targets focused on the COVID-19 outbreak. The results indicate a similarity between the emotional meaning of COVID-19 in society and the descriptive feeling rules, whereas the reported inner emotions were clearly distinct: Joy was irrelevant at the interpersonal and macrosocial levels but clearly important at the individual level. A mismatch also appeared for fear and hope. While fear was the most predominant emotion at the interpersonal and macrosocial levels during most of the phases, it was moderately predominant at the individual level. Hope followed the opposite pattern, being the most relevant emotion at the individual level but less relevant at the interpersonal and macrosocial levels. Each level might have different consequences: Mixed emotions at the individual level might promote resilience; fear perceived in other people might motivate protective behaviors; and sadness socially shared during Christmas might generate greater empathy. These results support the complexity of emotional concepts and the suitability of exploring them at different levels of analysis.

Deep Learning-Based Approach for Continuous Affect Prediction From Facial Expression Images in Valence-Arousal Space

Facial emotion recognition has attracted extensive attention from the affective computing community and several approaches have been proposed, mainly providing classification of facial expressions images using a set of discrete emotional labels. The quantification of human emotions from faces has been studied in a continuous 2D emotional space of valence and arousal, that describes the level of pleasantness and the intensity of excitement, respectively. Emotion assessment using valence-arousal computation is a challenging topic with several possible applications in health monitoring, e-learning, mental health diagnosis and monitoring of customer interest. Supervised learning of emotional valence-arousal for continuous affect prediction requires labeled data. However, annotation of facial images with the values for valence and arousal requires training human experts. In this paper, we propose a new and robust approach based on deep learning for continuous affect recognition and prediction. The novelty of our approach is that it maps the discrete labels and a learned facial expression representation to the continuous valence-arousal dimensional space. Given a discrete class of emotion and a facial image, our deep learning-based approach can predict the valence and arousal values accurately. Our proposed approach outperforms existing approaches for arousal and valence prediction on AffectNet dataset and it shows an impressive generalization ability on an unseen dataset for valence prediction.

Speech Driven Talking Face Generation From a Single Image and an Emotion Condition

Visual emotion expression plays an important role in audiovisual speech communication. In this work, we propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. Specifically, we design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input to render a talking face video synchronized with the speech and expressing the conditioned emotion. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system outperforms a state-of-the-art baseline system. Subjective evaluation of visual emotion expression and video realness also demonstrates the superiority of the proposed system. Furthermore, we conduct a human emotion recognition pilot study using generated videos with mismatched emotions among the audio and visual modalities. Results show that humans respond to the visual modality more significantly than the audio modality on this task.

Localized task-invariant emotional valence encoding revealed by intracranial recordings.

The ability to distinguish between negative, positive and neutral valence is a key part of emotion perception. Emotional valence has conceptual meaning that supersedes any particular type of stimulus, although it is typically captured experimentally in association with particular tasks. We sought to identify neural encoding for task-invariant emotional valence. We evaluated whether high-gamma responses (HGRs) to visually displayed words conveying emotions could be used to decode emotional valence from HGRs to facial expressions. Intracranial electroencephalography was recorded from 14 individuals while they participated in two tasks, one involving reading words with positive, negative, and neutral valence, and the other involving viewing faces with positive, negative, and neutral facial expressions. Quadratic discriminant analysis was used to identify information in the HGR that differentiates the three emotion conditions. A classifier was trained on the emotional valence labels from one task and was cross-validated on data from the same task (within-task classifier) as well as the other task (between-task classifier). Emotional valence could be decoded in the left medial orbitofrontal cortex and middle temporal gyrus, both using within-task classifiers and between-task classifiers. These observations suggest the presence of task-independent emotional valence information in the signals from these regions.