Emotion Recognition Processing Research Articles

Development of Emotion Recognition from Facial Expressions with Different Eye and Mouth Cues in Japanese People

Research has reported that Japanese people are more likely to focus on and look longer at eyes when reading emotions from facial expressions than their western counterparts. However, how these tendencies develop and whether there is a relationship between the two tendencies (to focus on the eyes and to look longer at the eyes) is unclear. The present study examined emotion recognition and gaze patterns in Japanese preschool children (n = 51) and university students (n = 57), using facial expressions with different eye and mouth cues. The results showed developmental changes in emotion recognition, with adults being more sensitive to negative emotions, whereas gaze patterns showed no developmental changes. Furthermore, there was no relationship between emotion recognition and gaze patterns. This suggests that the implicit and explicit processing of emotion recognition develops at different times, and that there is no direct relationship between the two processes.

Design of a team emotion forecast tool for bettering individual emotional well-being and team performance

Individual emotion can significantly impact one's well-being and a team's emotional environment can affect team performance. In addition, one's emotions in a team can affect others’ emotions. Thus, in order to have a positive team emotional state (environment) and avoid one's negative emotions propagating to others, a team emotion forecast tool is needed for teamwork management. How to design an easy-to-use team emotion forecast tool is our research question. In this study, we first propose to use keystroke and mouse dynamics as potentially effective signals to recognize emotions because of their ubiquity in the work environment, especially in remote work. And the emotion recognition process is non-intrusive and unobtrusive, which helps to identify emotions more objectively. Second, the user-centered design method is used to design such a tool/system. Third, a qualitative study has been conducted to analyze users’ demands and concerns. At last, a digital prototype has been designed and developed.

Modified Earthworm Optimization With Deep Learning Assisted Emotion Recognition for Human Computer Interface

Among the most prominent field in the human-computer interface (HCI) is emotion recognition using facial expressions. Posed variations, facial accessories, and non-uniform illuminations are some of the difficulties in the emotion recognition field. Emotion detection with the help of traditional methods has the shortcoming of mutual optimization of feature extraction and classification. Computer vision (CV) technology improves HCI by visualizing the natural world in a digital platform like the human brain. In CV technique, advances in machine learning and artificial intelligence result in further enhancements and changes, which ensures an improved and more stable visualization. This study develops a new Modified Earthworm Optimization with Deep Learning Assisted Emotion Recognition (MEWODL-ER) for HCI applications. The presented MEWODL-ER technique intends to categorize different kinds of emotions that exist in the HCI applications. To do so, the presented MEWODL-ER technique employs the GoogleNet model to extract feature vectors and the hyperparameter tuning process is performed via the MEWO algorithm. The design of automated hyperparameter adjustment using the MEWO algorithm helps in attaining an improved emotion recognition process. Finally, the quantum autoencoder (QAE) model is implemented for the identification and classification of emotions related to the HCI applications. To exhibit the enhanced recognition results of the MEWODL-ER approach, a wide-ranging simulation analysis is performed. The experimental values indicated that the MEWODL-ER technique accomplishes promising performance over other models with maximum accuracy of 98.91%.

Deep Residual Adaptive Neural Network Based Feature Extraction for Cognitive Computing with Multimodal Sentiment Sensing and Emotion Recognition Process

For the healthcare framework, automatic recognition of patients’ emotions is considered to be a good facilitator. Feedback about the status of patients and satisfaction levels can be provided automatically to the stakeholders of the healthcare industry. Multimodal sentiment analysis of human is considered as the attractive and hot topic of research in artificial intelligence (AI) and is the much finer classification issue which differs from other classification issues. In cognitive science, as emotional processing procedure has inspired more, the abilities of both binary and multi-classification tasks are enhanced by splitting complex issues to simpler ones which can be handled more easily. This article proposes an automated audio-visual emotional recognition model for a healthcare industry. The model uses Deep Residual Adaptive Neural Network (DeepResANNet) for feature extraction where the scores are computed based on the differences between feature and class values of adjacent instances. Based on the output of feature extraction, positive and negative sub-nets are trained separately by the fusion module thereby improving accuracy. The proposed method is extensively evaluated using eNTERFACE’05, BAUM-2 and MOSI databases by comparing with three standard methods in terms of various parameters. As a result, DeepResANNet method achieves 97.9% of accuracy, 51.5% of RMSE, 42.5% of RAE and 44.9%of MAE in 78.9sec for eNTERFACE’05 dataset. For BAUM-2 dataset, this model achieves 94.5% of accuracy, 46.9% of RMSE, 42.9%of RAE and 30.2% MAE in 78.9 sec. By utilizing MOSI dataset, this model achieves 82.9% of accuracy, 51.2% of RMSE, 40.1% of RAE and 37.6% of MAE in 69.2sec. By analysing all these three databases, eNTERFACE’05 is best in terms of accuracy achieving 97.9%. BAUM-2 is best in terms of error rate as it achieved 30.2 % of MAE and 46.9% of RMSE. Finally MOSI is best in terms of RAE and minimal response time by achieving 40.1% of RAE in 69.2 sec.

Emotional Response using Power Spectrum Approach

The objective of this study is to detect affective response of children to facial expression based on alpha power density of brain activity. Electroencephalography data were collected from 10 typical children. The alpha power temporal information of active brain regions was extracted. Performance of the power spectrum feature was evaluated in emotion recognition process using K nearest neighbor, a regularized least square and multilayer perceptron classifier. A statistical analysis indicated right alpha activity during negative and calm emotional states. Statistical results showed significant difference between rest conditions and emotional state. The best accuracy we got to detect emotional states is by using regularized least square that is 70%.

Students’ knowledge on emotion expression and recognition in computer-mediated communication: A comparative case study

In this study, we explore processes of emotion expression and recognition in computer-mediated civic discussions in small groups. We ask what generates processes of emotion expression and recognition in digital discussions, what constitutes the emotion expression and recognition practices used in such contexts, and in what ways these practices are affected by medium. Ninety-eight participants took part in both written and video discussions and filled out questionnaires concerning their emotional experiences. Some of these participants were interviewed. A two-tiered thematic and linguistic analysis was applied to the data collected from the discussions and the interviews (totaling 7139 utterances). Findings reveal four generators giving rise to discussants' emotion expression and recognition processes, including firmness of the discussants' opinions and their cultural and religious background. Practices of emotion expression and recognition appear to vary substantially between written and face-to-face settings. For example, in the latter, participants rely heavily on tone of voice and facial expressions when recognizing others’ emotions, while in the former, because of the lack of non-verbal cues, discussants assign more weight to their own emotions, experiences, and background. Moreover, participants choose their words more carefully, and are more likely to delay their response or to remain silent when expressing their emotions in writing.

Transcranial Direct Current Stimulation Over the Right Temporal Parietal Junction Facilitates Spontaneous Micro-Expression Recognition.

Micro-expressions are fleeting and subtle emotional expressions. As they are spontaneous and uncontrollable by one’s mind, micro-expressions are considered an indicator of genuine emotions. Their accurate recognition and interpretation promote interpersonal interaction and social communication. Therefore, enhancing the ability to recognize micro-expressions has captured much attention. In the current study, we investigated the effects of training on micro-expression recognition with a Chinese version of the Micro-Expression Training Tool (METT). Our goal was to confirm whether the recognition accuracy of spontaneous micro-expressions could be improved through training and brain stimulation. Since the right temporal parietal junction (rTPJ) has been shown to be involved in the explicit process of facial emotion recognition, we hypothesized that the rTPJ would play a role in facilitating the recognition of micro-expressions. The results showed that anodal transcranial direct-current stimulation (tDCS) of the rTPJ indeed improved the recognition of spontaneous micro-expressions, especially for those associated with fear. The improved accuracy of recognizing fear spontaneous micro-expressions was positively correlated with personal distress in the anodal group but not in the sham group. Our study supports that the combined use of tDCS and METT can be a viable way to train and enhance micro-expression recognition.

Is Emotion recognition processing across menstrual cycle and a history of Postpartum Depression potential risk factors for Premenstrual Dysphonic Disorder?

IntroductionMany women during the different phases of the menstrual cycle experience significant emotional and cognitive changes; for some, these changes can affect their everyday living. Premenstrual Dysphonic Disorder (PMDD) is a health problem that affects up to 5% of women of childbearing age. The exact cause is unknown; still, hormonal changes throughout the menstrual cycle may play a role. Women with a family history of Postpartum Depression (PPD) may be at increased risk.ObjectivesThe purpose was to examine if Emotion recognition processing across menstrual cycle and a history of PPD are potential risk factors for PMDD.MethodsWe identified 34 women with a history of PPD and contacted their daughters to explain the purpose of our study. Of those meeting the criteria to participate, 38 volunteered (aged 18-30 y.o., right handed, educational level >9 y., regular cycle duration). The Emotion Recognition Task (ERT) was administered in the luteal and the follicular phase. Women found to present significant differences in emotion recognition depending on the estradiol and progesterone levels were clinically interviewed (DSM-V).ResultsOf the 16 women who have showed significant differences across the two phases of the menstrual cycle, 7 were diagnosed with PMDD (43,7%). Among the ones who have not presented differences (22), only 2 received a diagnosis of PMDD (9%).ConclusionsThis study shows that Emotion recognition processing across menstrual cycle and a history of PPD may predict which women could be at risk for PMDD, playing, therefore a key role in PMDD early diagnosis.DisclosureNo significant relationships.

Do women with Premenstrual Dysphonic Disorder experience differences in emotion recognition during the menstrual cycle?

IntroductionThe relationship between behavioral changes and the menstrual cycle in women at a reproductive age has been investigated in several studies; women during every menstrual cycle experience noticeable changes in levels of sex hormones which are consequently reflected on their mood and behavior. The relationship between the menstrual cycle and the emotion recognition processing has been also studied.ObjectivesThe aim of this study was to examine if differences exist between women with Premenstrual Dysphonic Disorder (PMDD) and women without PMDD in Emotion recognition processing across menstrual cycle.MethodsWe examined 26 women with a PMDD and 30 women without PMDD, who have both visited the Mental Health Centre (aged 18-35 y.o., right handed, educational level >9 y., regular cycle duration). Women were clinically interviewed (DSM-V); also the Emotion Recognition Task (ERT) was administered in the luteal and the follicular phase.ResultsWomen with PMDD showed significant differences in emotion recognition depending on the the luteal and the follicular phase (according to estradiol and progesterone level) whereas women without PMDD did not present significantly different responses to the emotional stimuli.ConclusionsOur findings suggest that there is an effect of PMDD on emotional facial recognition across the two phases of the menstrual cycle. Thus, the importance of incorporating ERT in the formal clinical examination of PMDD is highlighted.DisclosureNo significant relationships.

Dopaminergic Modulation of Dynamic Emotion Perception.

Emotion recognition abilities are fundamental to our everyday social interaction. A large number of clinical populations show impairments in this domain, with emotion recognition atypicalities being particularly prevalent among disorders exhibiting a dopamine system disruption (e.g., Parkinson's disease). Although this suggests a role for dopamine in emotion recognition, studies employing dopamine manipulation in healthy volunteers have exhibited mixed neural findings and no behavioral modulation. Interestingly, while a dependence of dopaminergic drug effects on individual baseline dopamine function has been well established in other cognitive domains, the emotion recognition literature so far has failed to account for these possible interindividual differences. The present within-subjects study therefore tested the effects of the dopamine D2 antagonist haloperidol on emotion recognition from dynamic, whole-body stimuli while accounting for interindividual differences in baseline dopamine. A total of 33 healthy male and female adults rated emotional point-light walkers (PLWs) once after ingestion of 2.5 mg haloperidol and once after placebo. To evaluate potential mechanistic pathways of the dopaminergic modulation of emotion recognition, participants also performed motoric and counting-based indices of temporal processing. Confirming our hypotheses, effects of haloperidol on emotion recognition depended on baseline dopamine function, where individuals with low baseline dopamine showed enhanced, and those with high baseline dopamine decreased emotion recognition. Drug effects on emotion recognition were related to drug effects on movement-based and explicit timing mechanisms, indicating possible mediating effects of temporal processing. Results highlight the need for future studies to account for baseline dopamine and suggest putative mechanisms underlying the dopaminergic modulation of emotion recognition.SIGNIFICANCE STATEMENT A high prevalence of emotion recognition difficulties among clinical conditions where the dopamine system is affected suggests an involvement of dopamine in emotion recognition processes. However, previous psychopharmacological studies seeking to confirm this role in healthy volunteers thus far have failed to establish whether dopamine affects emotion recognition and lack mechanistic insights. The present study uncovered effects of dopamine on emotion recognition in healthy individuals by controlling for interindividual differences in baseline dopamine function and investigated potential mechanistic pathways via which dopamine may modulate emotion recognition. Our findings suggest that dopamine may influence emotion recognition via its effects on temporal processing, providing new directions for future research on typical and atypical emotion recognition.

Impact of Feature Extraction and Feature Selection Algorithms on Punjabi Speech Emotion Recognition Using Convolutional Neural Network

As a challenge to refine the spontaneity and productivity of a machine and human coherence, speech emotion recognition has been an overriding area of research. The trustability and fulfillment of emotion recognition are largely involved with the feature extraction and selection processes. An important role is played in exploring and distinguishing audio content during the feature extraction phase. Also, the features that have been extracted should be resilient to a number of disturbances and reliable enough for an adequate classification system. This article focuses on three main components of a Speech Emotion Recognition (SER) process. The first one is the optimal feature extraction method for a Punjabi SER system. The second one is the use of an appropriate feature selection method that selects effectual features from the ones extracted in the first step and removes the redundant features to improve the conduct of emotion recognition. The third one is the classification model that has been used further for emotion recognition. So the scope of this article is to explain the three main steps of the Punjabi SER system: feature extraction, feature selection, and emotion recognition with classifier. The results have been calculated and compared for number of feature set combinations, with and without a feature selection process. A total of 10 experiments are carried out, and various performance metrics such as precision, recall, F1-score, accuracy, and so on, are used to demonstrate the results.

Data-driven affective product design using complete three-dimensional surface data

As the main part of design display and evaluation, product three-dimensional (3D) form is the core object in affective product design. However, previous research has not yet addressed the development of technical models and method involving complete 3D surface data, and thus cannot guarantee the quality of affective product design. By using the techniques of triangular mesh model, spherical harmonic and conditional variational auto-encoder, this paper proposes a data-driven affective product design method composed of several technical models using complete 3D surface data. These models include: mathematical model for quantifying 3D form, recognition model for recognizing customer’s affective responses, and generative model for generating new 3D forms. For affective product design, the mathematical model achieves the acquisition and processing of complete 3D surface data, the recognition model improves the objectivity and accuracy of recognition by integrating the 3D form data into the calculation process of emotion recognition, and the generative model realizes the automatic generation of new 3D forms in response to emotional data based on the recognition results. Each model provides technical support for realizing the acquisition, processing and generation of complete 3D surface data of product form, and ensures the systematicness and completeness of the proposed method for the affective product design involving 3D form innovation. The feasibility of the method is verified by an example of car design, and the results show that it is an effective affective product design method involving 3D form innovation.

A Review on Emotion Recognition with Machine Learning Using EEG Signals

Emotions are critical in people's daily lives since their decision making, interaction, intelligence, and perception are all influenced by the emotions they display. Emotion recognition with machine learning based on EEG signals has been an exciting topic and employed in several areas, such as health care, social security, and safe driving. In this paper, a review on emotion recognition using EEG signals employing machine learning is carried out based on various factors, such as the stimulus used, equipment, modalities, filters, features, classifiers, and detected emotions, along with the limitations. This paper identifies the basic methodology used in the emotion recognition process with various tools and technologies utilized in it. Finally, it gives the issues and challenges for future research directions.

Deep Learning-Based Approach for Emotion Recognition Using Electroencephalography (EEG) Signals Using Bi-Directional Long Short-Term Memory (Bi-LSTM).

Emotions are an essential part of daily human communication. The emotional states and dynamics of the brain can be linked by electroencephalography (EEG) signals that can be used by the Brain–Computer Interface (BCI), to provide better human–machine interactions. Several studies have been conducted in the field of emotion recognition. However, one of the most important issues facing the emotion recognition process, using EEG signals, is the accuracy of recognition. This paper proposes a deep learning-based approach for emotion recognition through EEG signals, which includes data selection, feature extraction, feature selection and classification phases. This research serves the medical field, as the emotion recognition model helps diagnose psychological and behavioral disorders. The research contributes to improving the performance of the emotion recognition model to obtain more accurate results, which, in turn, aids in making the correct medical decisions. A standard pre-processed Database of Emotion Analysis using Physiological signaling (DEAP) was used in this work. The statistical features, wavelet features, and Hurst exponent were extracted from the dataset. The feature selection task was implemented through the Binary Gray Wolf Optimizer. At the classification stage, the stacked bi-directional Long Short-Term Memory (Bi-LSTM) Model was used to recognize human emotions. In this paper, emotions are classified into three main classes: arousal, valence and liking. The proposed approach achieved high accuracy compared to the methods used in past studies, with an average accuracy of 99.45%, 96.87% and 99.68% of valence, arousal, and liking, respectively, which is considered a high performance for the emotion recognition model.

Oversampling Approach Using Radius-SMOTE for Imbalance Electroencephalography Datasets

Several studies related to emotion recognition based on Electroencephalogram signals have been carried out in feature extraction, feature representation, and classification. However, emotion recognition is strongly influenced by the distribution or balance of Electroencephalogram data. On the other hand, the limited data obtained significantly affects the imbalance condition of the resulting Electroencephalogram signal data. It has an impact on the low accuracy of emotion recognition. Therefore, based on these problems, the contribution of this research is to propose the Radius SMOTE method to overcome the imbalance of the DEAP dataset in the emotion recognition process. In addition to the EEG data oversampling process, there are several vital processes in emotion recognition based on EEG signals, including the feature extraction process and the emotion classification process. This study uses the Differential Entropy (DE) method in the EEG feature extraction process. The classification process in this study compares two classification methods, namely the Decision Tree method and the Convolutional Neural Network method. Based on the classification process using the Decision Tree method, the application of oversampling with the Radius SMOTE method resulted in the accuracy of recognizing arousal and valence emotions of 78.78% and 75.14%, respectively. Meanwhile, the Convolutional Neural Network method can accurately identify the arousal and valence emotions of 82.10% and 78.99%, respectively. Doi: 10.28991/ESJ-2022-06-02-013 Full Text: PDF

The Importance of Being Familiar: The Role of Semantic Knowledge in the Activation of Emotions and Factual Knowledge from Music in the Semantic Variant of Primary Progressive Aphasia.

The role of semantic knowledge in emotion recognition remains poorly understood. The semantic variant of primary progressive aphasia (svPPA) is a degenerative disorder characterized by progressive loss of semantic knowledge, while other cognitive abilities remain spared, at least in the early stages of the disease. The syndrome is therefore a reliable clinical model of semantic impairment allowing for testing the propositions made in theoretical models of emotion recognition. The main goal of this study was to investigate the role of semantic memory in the recognition of basic emotions conveyed by music in individuals with svPPA. The performance of 9 individuals with svPPA was compared to that of 32 control participants in tasks designed to investigate the ability: a) to differentiate between familiar and non-familiar musical excerpts, b) to associate semantic concepts to musical excerpts, and c) to recognize basic emotions conveyed by music. Results revealed that individuals with svPPA showed preserved abilities to recognize familiar musical excerpts but impaired performance on the two other tasks. Moreover, recognition of basic emotions and association of musical excerpts with semantic concepts was significantly better for familiar than non-familiar musical excerpts in participants with svPPA. Results of this study have important implications for theoretical models of emotion recognition and music processing. They suggest that impairment of semantic memory in svPPA affects both the activation of emotions and factual knowledge from music and that this impairment is modulated by familiarity with musical tunes.

Augmented Reality Beauty Apps for Physical Appearance and Attractiveness: Face and Body Emotion Recognition Processes, Negative Mood and Self-Esteem, and Appealing Self-Presentation

Augmented Reality Beauty Apps for Physical Appearance and Attractiveness: Face and Body Emotion Recognition Processes, Negative Mood and Self-Esteem, and Appealing Self-Presentation

Morphing Task: The Emotion Recognition Process in Children with Attention Deficit Hyperactivity Disorder and Autism Spectrum Disorder.

Recognizing a person’s identity is a fundamental social ability; facial expressions, in particular, are extremely important in social cognition. Individuals affected by autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) display impairment in the recognition of emotions and, consequently, in recognizing expressions related to emotions, and even their identity. The aim of our study was to compare the performance of participants with ADHD, ASD, and typical development (TD) with regard to both accuracy and speed in the morphing task and to determine whether the use of pictures of digitized cartoon faces could significantly facilitate the process of emotion recognition in ASD patients (particularly for disgust). This study investigated the emotion recognition process through the use of dynamic pictures (human faces vs. cartoon faces) created with the morphing technique in three pediatric populations (7–12 years old): ADHD patients, ASD patients, and an age-matched control sample (TD). The Chi-square test was used to compare response latency and accuracy between the three groups in order to determine if there were statistically significant differences (p < 0.05) in the recognition of basic emotions. The results demonstrated a faster response time in neurotypical children compared to ASD and ADHD children, with ADHD participants performing better than ASD participants on the same task. The overall accuracy parameter between the ADHD and ASD groups did not significantly differ.

Neural network modeling of altered facial expression recognition in autism spectrum disorders based on predictive processing framework

The mechanism underlying the emergence of emotional categories from visual facial expression information during the developmental process is largely unknown. Therefore, this study proposes a system-level explanation for understanding the facial emotion recognition process and its alteration in autism spectrum disorder (ASD) from the perspective of predictive processing theory. Predictive processing for facial emotion recognition was implemented as a hierarchical recurrent neural network (RNN). The RNNs were trained to predict the dynamic changes of facial expression movies for six basic emotions without explicit emotion labels as a developmental learning process, and were evaluated by the performance of recognizing unseen facial expressions for the test phase. In addition, the causal relationship between the network characteristics assumed in ASD and ASD-like cognition was investigated. After the developmental learning process, emotional clusters emerged in the natural course of self-organization in higher-level neurons, even though emotional labels were not explicitly instructed. In addition, the network successfully recognized unseen test facial sequences by adjusting higher-level activity through the process of minimizing precision-weighted prediction error. In contrast, the network simulating altered intrinsic neural excitability demonstrated reduced generalization capability and impaired emotional clustering in higher-level neurons. Consistent with previous findings from human behavioral studies, an excessive precision estimation of noisy details underlies this ASD-like cognition. These results support the idea that impaired facial emotion recognition in ASD can be explained by altered predictive processing, and provide possible insight for investigating the neurophysiological basis of affective contact.

Recognition Characteristics of Facial and Bodily Expressions: Evidence From ERPs.

In the natural environment, facial and bodily expressions influence each other. Previous research has shown that bodily expressions significantly influence the perception of facial expressions. However, little is known about the cognitive processing of facial and bodily emotional expressions and its temporal characteristics. Therefore, this study presented facial and bodily expressions, both separately and together, to examine the electrophysiological mechanism of emotional recognition using event-related potential (ERP). Participants assessed the emotions of facial and bodily expressions that varied by valence (positive/negative) and consistency (matching/non-matching emotions). The results showed that bodily expressions induced a more positive P1 component and a shortened latency, whereas facial expressions triggered a more negative N170 and prolonged latency. Among N2 and P3, N2 was more sensitive to inconsistent emotional information and P3 was more sensitive to consistent emotional information. The cognitive processing of facial and bodily expressions had distinctive integrating features, with the interaction occurring in the early stage (N170). The results of the study highlight the importance of facial and bodily expressions in the cognitive processing of emotion recognition.