Improve Emotion Recognition Research Articles

Exploiting Transfer Learning for Emotion Recognition Under Cloud-Edge-Client Collaborations

Emerging virtual reality/augmented reality games and self-driving cars necessitate accurate/responsive/private emotion recognition. Usually, traditional emotion recognition models are deployed at central servers, which results in the lack of abilities in generalization and covering the individual variation of clients. This paper proposes a responsive, localized, and private transfer learning based emotion recognition framework under the cloud-edge-client collaborations. Additionally, a 3-dimensional channel mapping method is designed to aggregate features extracted from electroencephalogram (EEG) signals for the generic emotion recognition model, which is further localized and personalized using transfer learning. Simulation results validate the performance of the proposed TLER framework in reducing model training time and improving emotion recognition accuracy.

Adapting the Interplay Between Personalized and Generalized Affect Recognition Based on an Unsupervised Neural Framework

Recent emotion recognition models, most of them being based on strongly supervised deep learning solutions, are rather successful in recognizing instantaneous emotion expressions. However, when applied to continuous interactions, these models show a weaker adaptation to a person-specific and long-term emotion appraisal. In this article, we present an unsupervised neural framework that improves emotion recognition by learning how to describe continuous affective behavior of individual persons. Our framework is composed of three self-organizing mechanisms: (1) a recurrent growing layer to cluster general emotion expressions, (2) a set of associative layers, acting as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">affective memories</i> to model specific emotional behavior of individual persons, (3) and an online learning layer which provides contextual modeling of continuous emotion expressions. We propose different learning strategies to integrate all three mechanisms and to improve the performance on arousal and valence recognition of the OMG-Emotion dataset. We evaluate our model with a series of experiments ranging from ablation studies assessing the different contributions of each neural component to an objective comparison with state-of-the-art solutions. The results from the evaluations show a good performance on emotion recognition of continuous emotions on monologue videos. Furthermore, we discuss how the model self-regulates the interplay between generalized and personalized emotion perception and how this influences the model’s reliability when recognizing unseen emotion expressions.

Dynamic Facial Expression Generation on Hilbert Hypersphere With Conditional Wasserstein Generative Adversarial Nets.

In this work, we propose a novel approach for generating videos of the six basic facial expressions given a neutral face image. We propose to exploit the face geometry by modeling the facial landmarks motion as curves encoded as points on a hypersphere. By proposing a conditional version of manifold-valued Wasserstein generative adversarial network (GAN) for motion generation on the hypersphere, we learn the distribution of facial expression dynamics of different classes, from which we synthesize new facial expression motions. The resulting motions can be transformed to sequences of landmarks and then to images sequences by editing the texture information using another conditional Generative Adversarial Network. To the best of our knowledge, this is the first work that explores manifold-valued representations with GAN to address the problem of dynamic facial expression generation. We evaluate our proposed approach both quantitatively and qualitatively on two public datasets; Oulu-CASIA and MUG Facial Expression. Our experimental results demonstrate the effectiveness of our approach in generating realistic videos with continuous motion, realistic appearance and identity preservation. We also show the efficiency of our framework for dynamic facial expressions generation, dynamic facial expression transfer and data augmentation for training improved emotion recognition models.

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

Speech emotion recognition (SER) has recently been receiving increased interest due to the rapid advancements in affective computing and human computer interaction. English, German, Mandarin and Indian are among the most commonly considered languages for SER along with other European and Asian languages. However, few researches have implemented Arabic SER systems due to the scarcity of available Arabic speech emotion databases. Although Egyptian Arabic is considered one of the most widely spoken and understood Arabic dialects in the Middle East, no Egyptian Arabic speech emotion database has yet been devised. In this work, a semi-natural Egyptian Arabic speech emotion (EYASE) database is introduced that has been created from an award winning Egyptian TV series. The EYASE database includes utterances from 3 male and 3 female professional actors considering four emotions: angry, happy, neutral and sad. Prosodic, spectral and wavelet features are computed from the EYASE database for emotion recognition. In addition to the classical pitch, intensity, formants and Mel-frequency cepstral coefficients (MFCC) widely implemented for SER, long-term average spectrum (LTAS) and wavelet parameters are also considered in this work. Speaker independent and speaker dependent experiments were performed for three different cases: (1) emotion vs. neutral classifications, (2) arousal and valence classifications and (3) multi-emotion classifications. Several analysis were made to explore different aspects related to Arabic SER including the effect of gender and culture on SER. Furthermore, feature ranking was performed to evaluate the relevance of the LTAS and wavelet features for SER, in comparison to the more widely used prosodic and spectral features. Moreover, anger detection performance is compared for different combinations of the implemented prosodic, spectral and wavelet features. Feature ranking and anger detection performance analysis showed that both LTAS and wavelet features were relevant for Arabic SER and that they significantly improved emotion recognition rates.

Improving emotion recognition in anorexia nervosa: An experimental proof-of-concept study.

Previous research has found increasing evidence for difficulties in emotion recognition ability (ERA) and social cognition in anorexia nervosa (AN), and recent models consider these factors to contribute to the development and maintenance of the disorder. However, there is a lack of experimental studies testing this hypothesis. Therefore, the present proof-of-concept study examined whether ERA can be improved by a single session of a computerized training in AN, and whether this has short-term effects on eating disorder symptoms. Forty inpatients (22.20 ± 7.15 years) with AN were randomly assigned to receive a single session of computerized training of ERA (TERA) or a sham training (training the recognition of different types of clouds). ERA, self-reported eating disorder symptoms, and body mass index (BMI) were assessed within 3 days before and after training. After training, both groups showed improved ERA, reduced self-reported eating disorder symptoms, and an increased BMI. As compared to patients in the control group, patients who received TERA showed greater improvements in ERA and self-reported eating disorder symptoms. ERA can be effectively trained in patients with AN. Moreover, short-term improvements in self-reported eating disorder symptoms provide tentative support for the hypothesis that difficulties in ERA contribute to the maintenance of AN, and that specific trainings of ERA hold promise as an additional component in AN treatment. Future studies are needed to replicate these findings in larger samples, and to investigate long-term effects and transfer into real-world settings.

Multi-scale Entropy and Multiclass Fisher’s Linear Discriminant for Emotion Recognition Based on Multimodal Signal

Emotion recognition using physiological signals has been a special topic frequently discussed by researchers and practitioners in the past decade. However, the use of SpO2 and Pulse rate signals for emotion recognitionisvery limited and the results still showed low accuracy. It is due to the low complexity of SpO2 and Pulse rate signals characteristics. Therefore, this study proposes a Multiscale Entropy and Multiclass Fisher’s Linear Discriminant Analysis for feature extraction and dimensional reduction of these physiological signals for improving emotion recognition accuracy in elders. In this study, the dimensional reduction process was grouped into three experimental schemes, namely a dimensional reduction using only SpO2 signals, pulse rate signals, and multimodal signals (a combination feature vectors of SpO2 and Pulse rate signals). The three schemes were then classified into three emotion classes (happy, sad, and angry emotions) using Support Vector Machine and Linear Discriminant Analysis Methods. The results showed that Support Vector Machine with the third scheme achieved optimal performance with an accuracy score of 95.24%. This result showed a significant increase of more than 22%from the previous works.

Semi-Supervised Learning for Continuous Emotion Recognition Based on Metric Learning

Emotion recognition is important for the interaction between human and artificial intelligence. Recently, the performance of facial image-based emotion recognition has been improved with deep learning's power. Nonetheless, huge data and label information for training are burdensome. In particular, annotating emotion labels in the continuous domain is very costly. Thus, we propose a novel semi-supervised learning that can not only reduce the annotation cost, but also improve emotion recognition performance by training with additional unlabeled data. The proposed method employs deep metric learning to improve feature embedding performance. Also, pseudo labels of unlabeled data are produced by analyzing inter-data distance in the feature space. Since pseudo labeling makes unlabeled data trainable, it increases overall performance. The experimental results show that the proposed method provides outstanding performance in the well-known MAHNOB-HCI dataset and the INHA dataset produced by our research team.

Frontal EEG Asymmetry of Emotion for the Same Auditory Stimulus

Emotions play an important role in human interaction and decision-making processes. Frontal asymmetry in brain activity is a promising neurophysiological indicator of emotion. Emotions are psychologically explained by the valence-arousal model, but as yet, frontal asymmetry has not been fully explained by this model. In this study, we explored frontal asymmetry of emotions based on the valence-arousal model using the same auditory stimulus. Changes in emotional states using self-report questionnaires were investigated before and after the auditory stimulus. Spectral power and weighted phase lag index were calculated in the delta, theta, alpha, beta, and gamma bands. Phase-amplitude coupling was also measured to explore communication among different frequency bands associated with emotions. After the auditory stimulus, alpha power decreased in both left and right frontal regions and the delta-weighted phase lag index in the left-right regions was increased. However, no frontal asymmetry was identified after the auditory stimulus. Additionally, we explored the brain changes according to the valence-arousal model based on emotional states. After the auditory stimulus, frontal asymmetry of alpha power was clearly observed only for negative valence. This finding was possible because subjective emotions were considered despite listening to the same stimulus. Finally, phase-amplitude coupling identified left-hemisphere dominance after the auditory stimulus, regardless of subjective emotions. These results may help us understand frontal asymmetry associated with emotional mechanisms. In addition, these findings can be used directly in the brain-computer interface to improve emotion recognition performance for real-world practical applications.

Rehabilitation of emotion recognition in Alzheimer's disease and study of the consequences on gaze strategy, behavior disorders and family caregivers’ burden

Backgrounds Emotion recognition is impaired in Alzheimer's disease (AD) [1] . Recently, we have highlighted that this kind of impairment is linked to atypical gaze strategy and behavior disorders [2] . Neuropsychiatric disorders are major factors of caregiver's burden, much more than cognitive impairment [3] . A positive impact of the training of affect recognition (TAR) has been found in AD [4] . This emotion recognition program is based on error-less learning, over-learning, immediate positive feedback and abstraction of facial features. This research aimed to confirm the improvement of facial emotion recognition performance in AD by using the TAR program, and aimed to study the consequences on gaze strategies, behavior disorders and caregiver's burden. Methods Four couples of caregivers-patients between 56–88 years old, with AD (mild and mild-moderate stage), were recruited. Participants with AD received 12 sessions of TAR (2 per week). Emotion recognition was assessed by using 28 items of the Ekman faces library. Eye movements were recorded with a Mobile EyeBrain Tracker®. The caregivers completed the ‘NeuroPsychiatric Inventory’ (NPI) to measure behavioral disorders and the ‘Zarit’ to measure the level of burden. Assessments were conducted before and after rehabilitation. Results A positive impact of TAR is highlighted with 11% improvement in emotion recognition. Especially, for sadness (133% improvement) and fear (62.5% improvement). Modification of gaze strategies is observed: eye region for anger and mouth region for happiness are more explored. Finally, there is a decrease in NPI and Zarit scores. Conclusions Emotion recognition abilities can be improved in AD by modifying gaze strategies during emotion decoding. This kind of rehabilitation has beneficial effects on caregivers’ burden due to the decrease of behaviors disorders.

Speech Emotion Recognition Based on Gender Influence in Emotional Expression

The real challenge in human-computer interaction is understanding human emotions by machines and responding to it accordingly. Emotion varies by gender and age of the speaker, location, and cause. This article focuses on the improvement of emotion recognition (ER) from speech using gender-biased influences in emotional expression. The problem is addressed by testing emotional speech with an appropriate specific-gender ER system. As acoustical characteristics vary among the genders, there may not be a common optimal feature set across both genders. Gender-based speech emotion recognition, a two-level hierarchical ER system is proposed, where the first level is gender identification which identifies the gender, and the second level is a gender-specific ER system, trained with an optimal feature set of expressions of a particular gender. The proposed system increases the accuracy of traditional Speech Emotion Recognition Systems (SER) by 10.36% than the SER trained with mixed gender training when tested on the EMO-DB Corpus.

Improved emotion recognition in Spanish social media through incorporation of lexical knowledge

Emotions play an important role in human intelligence and behaviour and are a major vehicle for communication. Therefore, the integration of emotions in computational models can improve the human–computer interaction systems. In this paper, we present a study of different machine learning approaches to automatically recognise emotions in messages written in Spanish on social media. Although the computational treatment of emotion is more difficult than other sentiment analysis tasks, the baseline of some machine learning algorithms achieve an acceptable accuracy showing that it is possible to tackle the problem using some basic natural language processing techniques. In this study we have experimented with the integration of knowledge from different affective lexical resources. We conclude that the incorporation of lexical affective features leads to improvement over most baseline figures with significant improvement. Indeed, we observe that the use of resources generated particularly for emotion recognition in other languages than English is a promising approach to enhance basic machine learning systems. Particularly, we used a Spanish lexical resource and we notice that it always improves the results. In the best case, it improves 6.15% of the results obtained using the Naive Bayes classifier.

Emotional functioning and the development of internalizing and externalizing problems in young boys with and without autism spectrum disorder.

Children with autism spectrum disorder are at risk of developing internalizing and externalizing problems. However, information on early development of behavior problems and the contributing role of emotional functioning in preschool children with autism spectrum disorder is scarce. This study collected data of boys with and without autism spectrum disorder (N = 156; age: 2–6 years) over three consecutive years (three waves), about their internalizing and externalizing symptoms and emotional functioning (i.e. emotion control, recognition, and vocabulary), using parent-report questionnaires. No age effect was found on internalizing or externalizing problems for boys with and without autism spectrum disorder. Boys with autism spectrum disorder displayed more behavior problems than their typically developing peers and showed lower levels of emotional functioning. Better emotion control and improved emotion recognition were associated with a decrease in problem behaviors for boys with and without autism spectrum disorder, whereas improved emotion vocabulary was uniquely related to a decrease in externalizing problems in boys with autism spectrum disorder. Our findings suggest that boys with and without autism spectrum disorder showed similar developmental courses of internalizing and externalizing problems. However, lower levels of emotional functioning were already more pronounced in boys with autism spectrum disorder at a young age. This contributes to higher levels of behavior problems.

Randomized Controlled Trial of Adjunctive Social Cognition and Interaction Training, Adjunctive Therapeutic Alliance Focused Therapy, and Treatment As Usual Among Persons With Serious Mental Illness.

As one of the areas of greatest concern for people with serious mental illness (SMI) are unmet social needs, psychosocial interventions have been developed to address them. The current study utilized a randomized controlled trial to examine the effectiveness of social cognition and interaction training (SCIT) versus a therapeutic alliance focused theraphy (TAFT) versus a treatment-as-usual (TAU) control group on social functioning and quality of life as primary outcomes and social cognition variables as secondary outcomes. Sixty-three persons between the ages of 24 and 69 years with SMI (41 men and 22 women), completers of the trial (23 in SCIT, 20 in TAFT, and 20 in TAU), were assessed at baseline, completion, and at a 3-month follow-up with measurements assessing social cognition (The Facial Emotion Identification Task, The Faux pas test, The Ambiguous Intentions Hostility Questionnaire) social functioning, (The Social Skills Performance Assessment, The Wisconsin Social Quality of Life Scale), and therapeutic alliance (adapted version for group of system for observing family therapy alliance). Results reveal that the two interventions were more effective than the control condition (TAU) in reducing attribution bias anger scores, SCIT was also effective in improving theory of mind (as can be seen in Faux pas test scores), and the TAFT in improving emotion recognition and reducing intentionality attribution bias scores. Improvement was related to therapeutic alliance which did not differ between the two intervention groups. Considering the role of alliance, it is recommended to consider the integration of the two studied interventions with other approaches that emphasize alliance and reflection. Clinical Trial Registration: www.ClinicalTrials.gov, identifier NCT02380885.

Smartphone-based emotion recognition skills training for alexithymia - A randomized controlled pilot study

BackgroundNeurobiological studies suggest that deficits in emotion recognition are common phenomena in alexithymia. Thus, effective treatments for alexithymia often include skills training in the domain of emotion recognition. Given that smartphone-based interventions (SBIs) offering skills training have been shown to be promising adjuncts to psychological treatments, a blended SBI facilitating the training of emotional skills might be effective in reducing alexithymia. MethodsIn this pilot trial, N = 29 individuals reporting elevated alexithymia levels were randomly assigned to a blended SBI including a psychoeducation session and 14 days of training with the mindtastic alexithymia app (MT-ALEX) or a psychoeducation-only control condition. Primary outcome was emotion recognition skills as assessed in a computer-based two-choice task paradigm. ResultsOn average, participating in the SBI was associated with a significant increase in computer-assessed emotion recognition skills compared to the control condition (d = 0.97). ConclusionsStudy findings provide preliminary evidence that SBIs can improve emotion recognition skills in alexithymic individuals. Research using larger samples and targeting clinical populations is necessary to further evaluate the potential of MT-ALEX.

Improving emotion recognition in schizophrenia with "VOICES": An on-line prosodic self-training.

IntroductionEmotion recognition (ER) deficits have been extensively demonstrated in schizophrenia. These deficiencies are not only restricted to facial emotion recognition but also include emotional prosody (tone of the voice) recognition deficits. They have been significantly associated with symptom severity and poor social functioning. The aim of this study was to test the efficacy, in real clinical conditions, of an online self-training prosodic game included in the Social Cognition rehabilitation program e-Motional Training.MethodA randomized, single-blind multicenter clinical trial was conducted with 50 outpatients with schizophrenia or schizoaffective disorder. The control group was treated with Treatment-as-usual (TAU), based on drug therapy, case management and individual and group psychotherapy (not focused on Social Cognition). The intervention group was treated with TAU plus the employment of Voices, an online self-training program devised for prosodic rehabilitation.Statistical analysisLinear regression was used to evaluate the effectiveness of the intervention in emotion recognition measured with the Reading the Mind in the Voice–Spanish Version (RMV-SV).ResultsThere were statistically significant differences between the intervention and control group measured with RMV-SV (β = 3,6[IC 95%], p<0.001), with a response variable in RMV post R2 = 0,617.DiscussionVoices, a prosodic rehabilitation game included in e-Motional Training, seems to be a promising tool for improving emotional voice recognition deficits in schizophrenia, filling the need for such interventions.

Intranasal oxytocin modulates higher order social cognition in body dysmorphic disorder.

Body dysmorphic disorder (BDD) is a psychiatric disorder with specific impairments in social cognition related to excessive concerns about one's appearance. Individuals with BDD have difficulty identifying emotional expressions and attribute internal factors for others' emotional expressions in self-referent (but not other-referent) scenarios. Given the role of oxytocin in regulating social approach behavior and social salience, we hypothesized that oxytocin would improve biases in emotion recognition, attributions, and threat interpretations in individuals with BDD, compared to healthy controls (HCs). This is the first study to examine the effects of oxytocin in people with BDD. Eighteen participants with BDD and 16 HCs received a single dose of 24 international units of intranasal oxytocin (Syntocinon®) or matching placebo in a randomized, placebo-controlled, within-subject crossover design. Participants completed the Emotion Recognition Task and Interpretation Questionnaire 45min after administration. Oxytocin, relative to placebo, did not improve emotion recognition accuracy in either self-referent or other-referent contexts for individuals with BDD. However, oxytocin led to greater internal attributions in other-referent contexts for those with BDD compared to HCs. Rather than reducing self-blame, oxytocin led to other-directed blame. Oxytocin did not impact threat interpretations in BDD. Oxytocin may only impact specific aspects of higher order social cognition in BDD and may have unwanted effects on emotion attributions. Our results caution its clinical use in BDD.

Multi-level region-based Convolutional Neural Network for image emotion classification

Analyzing emotional information of visual content has attracted growing attention for the tendency of internet users to share their feelings via images and videos online. In this paper, we investigate the problem of affective image analysis, which is very challenging due to its complexity and subjectivity. Previous research reveals that image emotion is related to low-level to high-level visual features from both global and local view, while most of the current approaches only focus on improving emotion recognition performance based on single-level visual features from a global view. Aiming to utilize different levels of visual features from both global and local view, we propose a multi-level region-based Convolutional Neural Network (CNN) framework to discover the sentimental response of local regions. We first employ Feature Pyramid Network (FPN) to extract multi-level deep representations. Then, an emotional region proposal method is used to generate proper local regions and remove excessive non-emotional regions for image emotion classification. Finally, to deal with the subjectivity in emotional labels, we propose a multi-task loss function to take the probabilities of images belonging to different emotion classes into consideration. Extensive experiments show that our method outperforms the state-of-the-art approaches on various commonly used benchmark datasets.

Providing a Four-layer Method Based on Deep Belief Network to Improve Emotion Recognition in Electroencephalography in Brain Signals.

Background:One of the fields of research in recent years that has been under focused is emotion recognition in electroencephalography (EEG) signals. This study provides a four-layer method to improve people's emotion recognition through these signals and deep belief neural networks.Methods:In this study, using DEAP dataset, a four-layer method is established, which includes (1) preprocessing, (2) extracting features, (3) dimension reduction, and (4) emotion identification and estimation. To find the optimal choice in some of the steps of these layers, three different tests have been conducted. The first is finding the perfect window in feature extraction section that resulted in superiority of Hamming window to the other windows. The second is choosing the most appropriate number of filter bank and the best result was 26. The third test was also emotion recognition that its accuracy was 92.93 for arousal dimension, 92.64 for valence dimension, 93.14 for dominance dimension in two-class experiment and 76.28 for the arousal, 74.83 for the valence, and 75.64 for dominance in three-class experiment.Results:The results of this method show an improvement of 12.34% and 7.74% in two- and three-class levels in the arousal dimension. This improvement in the valence is 12.77 and 8.52, respectively.Conclusion:The results show that the proposed method can be used to improve the accuracy of emotion recognition.

Emotion Recognition Based on Fusion of Local Cortical Activations and Dynamic Functional Networks Connectivity: An EEG Study

In this paper, we present a method to improve emotion recognition based on the fusion of local cortical activations and dynamic functional network patterns. We estimate the cortical activations using power spectral density (PSD) with the Burg autoregressive model. On the other hand, we estimate the functional connectivity networks by utilizing the phase locking value (PLV). The results of cortical activations and connectivity networks show different patterns across three emotions at all frequency bands. Similarly, the results of fusion significantly improve the classification rate in terms of accuracy, sensitivity, specificity and the area under the receiver operator characteristics curve (AROC), p <; 0.05. The average improvement with fusion in all evaluation metrics are 6.84% and 4.1% when compared to PSD and PLV alone, respectively. The results clearly demonstrate the advantage of fusion of cortical activations with dynamic functional networks for developing human-computer interaction system in real-world applications.

Using animated vehicles with real emotional faces to improve emotion recognition in Chinese children with autism spectrum disorder.

The objective of the present study was to conduct an intervention study which aimed to improve emotion recognition for Chinese children with ASD by using animated vehicles with real emotional faces. A total of 21 children participated in the current study; participants consisted of 14 children (2 girls) with a formal diagnosis of ASD and 7 typically developing children. Participants were measured on emotional vocabulary and situation-facial expression matching before and after the intervention. Results indicated that the intervention significantly improved ASD children’s emotion recognition compared to their pre-intervention scores. Our findings suggest that this emotional recognition intervention using animated vehicles (i.e. The Transporters) is an effective early intervention for Chinese children with ASD.