Emotion Recognition Accuracy Research Articles

Vocal emotion recognition in attention-deficit hyperactivity disorder: a meta-analysis

ABSTRACT There is debate within the literature as to whether emotion dysregulation (ED) in Attention-Deficit Hyperactivity Disorder (ADHD) reflects deviant attentional mechanisms or atypical perceptual emotion processing. Previous reviews have reliably examined the nature of facial, but not vocal, emotion recognition accuracy in ADHD. The present meta-analysis quantified vocal emotion recognition (VER) accuracy scores in ADHD and controls using robust variance estimation, gathered from 21 published and unpublished papers. Additional moderator analyses were carried out to determine whether the nature of VER accuracy in ADHD varied depending on emotion type. Findings revealed a medium effect size for the presence of VER deficits in ADHD, and moderator analyses showed VER accuracy in ADHD did not differ due to emotion type. These results support the theories which implicate the role of attentional mechanisms in driving VER deficits in ADHD. However, there is insufficient data within the behavioural VER literature to support the presence of emotion processing atypicalities in ADHD. Future neuro-imaging research could explore the interaction between attention and emotion processing in ADHD, taking into consideration ADHD subtypes and comorbidities.

Influence of child and adult faces with face masks on emotion perception and facial mimicry

Emotional mimicry, the imitation of others’ emotion expressions, is related to increased interpersonal closeness and better interaction quality. Yet, little research has focused on the effect of face masks on emotional mimicry and none on (masked) child faces. To address this gap, we conducted an online experiment (N = 235, German sample, adult perceivers). Masks reduced emotion recognition accuracy for all expressions, except in the case of anger in masked child faces, where perceived anger was even increased. Perceived interpersonal closeness was reduced for masked happy and sad faces. For both child and adult expressers, masks reduced facial mimicry of happy expressions, with no mask effects for sadness and anger expression. A stronger mask effect on facial happiness mimicry of child faces was mediated by the degree of emotion recognition accuracy. Smiles shown by masked children were not recognized well, likely due to the absence of wrinkles around the eyes in child faces. Independent of masks, sadness shown by children was mimicked even more strongly than when shown by adults. These results provide evidence for facial mimicry of child expressions by adult perceivers and show that the effects of face masks on emotion communication may vary when children wear them.

Role of Emotion Reactivity to Predict Facial Emotion Recognition Changes with Aging

ABSTRACT Emotional intelligence includes an assortment of factors related to emotion function. Such factors involve emotion recognition (in this case via facial expression), emotion trait, reactivity, and regulation. We aimed to investigate how the subjective appraisals of emotional intelligence (i.e. trait, reactivity, and regulation) are associated with objective emotion recognition accuracy, and how these associations differ between young and older adults. Data were extracted from the CamCAN dataset (189 adults: 57 young/118 older) from assessments measuring these emotion constructs. Using linear regression models, we found that greater negative reactivity was associated with better emotion recognition accuracy among older adults, though the pattern was opposite for young adults with the greatest difference in disgust and surprise recognition. Positive reactivity and depression level predicted surprise recognition, with the associations significantly differing between the age groups. The present findings suggest the level to which older and young adults react to emotional stimuli differentially predicts their ability to correctly identify facial emotion expressions. Older adults with higher negative reactivity may be able to integrate their negative emotions effectively in order to recognize other’s negative emotions more accurately. Alternatively, young adults may experience interference from negative reactivity, lowering their ability to recognize other’s negative emotions.

Facial Expressions Increase Emotion Recognition Clarity and Improve Warmth and Attractiveness on a Humanoid Robot without Adding the Uncanny Valley

With the rising impact of social robots, research on understanding user perceptions and preferences of these robots gains further importance, especially for emotional expressions. However, research on facial expressions and their effects on human-robot interaction has had mixed results. To unpack this issue further, we investigated users’ emotion recognition accuracy and perceptions when interacting with a social robot that displayed emotional facial expressions or not in a storytelling setting. In our experiment, twenty-eight participants received verbal feedback either with or without facial expressions from the robot. Participants showed a significant recognition accuracy effect for emotions and significantly higher clarity for disgust, happiness, and surprise in the facial expression condition than in the no facial expression condition. In addition, participants rated Milo with facial expressions significantly higher than Milo without facial expressions in warmth and attractiveness. No significant differences were found among the rating scores of naturalness. The results from the present study indicated the importance of facial expressions when considering design choices in social robots.

Cultural differences in recognizing emotions of masked faces.

In the wake of the global pandemic, interacting with others while wearing masks has emerged as a global challenge. A growing body of literature has reported that face masks hinder emotion recognition in Western populations. Given that diagnostic facial features for recognizing specific emotions (e.g., happiness) differ between Western and Eastern cultures, there may be cultural differences in the effects of face masks on emotion recognition. Relying on the previous findings showing cultural differences in emotion recognition, we conducted a preregistered study where 203 American and 209 Japanese participants judged the emotional expressions of faces (happy, fearful, angry, sad, disgust, and neutral) with and without masks. The results showed cultural differences in emotion recognition of faces with and without masks. Specifically, face masks decreased the accuracy of happy emotion recognition in the Americans but not in the Japanese. The results suggest that the effect of wearing masks on emotion recognition depends on the types of emotions and culture and supports previous findings indicating cultural differences in the decoding strategy for facial expressions. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Gleaning Emotions from Virtual Stickers: An Intercultural Study

Virtual stickers—as a newer generation of graphic icons—possess the potential to facilitate online intercultural communication through emotional expression. However, people from different cultural backgrounds rely on varied display and decoding rules, which lead to disagreement on the emotions communicated by stickers. Drawing from the dialect theory of communicating emotion, the current study employs an online survey to examine whether culture-, emotion-, and technology-related factors impact American college students’ recognition accuracy of 6 basic emotions expressed through 36 Chinese animated and static stickers. Results demonstrate that sticker use frequency, intercultural communication competence, and empathy are positively related to cross-cultural sticker emotion recognition accuracy. However, neither prior intercultural experience with Chinese students nor social media use frequency influences cross-cultural sticker emotion recognition accuracy. No gender differences were identified. This study represents one of the first to systematically examine predictors of sticker interpretation accuracy in a cross-cultural context.

Enhancing Speech Emotions Recognition Using Multivariate Functional Data Analysis

Speech Emotions Recognition (SER) has gained significant attention in the fields of human–computer interaction and speech processing. In this article, we present a novel approach to improve SER performance by interpreting the Mel Frequency Cepstral Coefficients (MFCC) as a multivariate functional data object, which accelerates learning while maintaining high accuracy. To treat MFCCs as functional data, we preprocess them as images and apply resizing techniques. By representing MFCCs as functional data, we leverage the temporal dynamics of speech, capturing essential emotional cues more effectively. Consequently, this enhancement significantly contributes to the learning process of SER methods without compromising performance. Subsequently, we employ a supervised learning model, specifically a functional Support Vector Machine (SVM), directly on the MFCC represented as functional data. This enables the utilization of the full functional information, allowing for more accurate emotion recognition. The proposed approach is rigorously evaluated on two distinct databases, EMO-DB and IEMOCAP, serving as benchmarks for SER evaluation. Our method demonstrates competitive results in terms of accuracy, showcasing its effectiveness in emotion recognition. Furthermore, our approach significantly reduces the learning time, making it computationally efficient and practical for real-world applications. In conclusion, our novel approach of treating MFCCs as multivariate functional data objects exhibits superior performance in SER tasks, delivering both improved accuracy and substantial time savings during the learning process. This advancement holds great potential for enhancing human–computer interaction and enabling more sophisticated emotion-aware applications.

Research on Speech Emotion Recognition Based on Teager Energy Operator Coefficients and Inverted MFCC Feature Fusion

As an important part of our daily life, speech has a great impact on the way people communicate. The Mel filter bank used in the extraction process of MFCC has a better ability to process the low-frequency component of a speech signal, but it weakens the emotional information contained in the high-frequency part of the speech signal. We used the inverted Mel filter bank to enhance the feature processing of the high-frequency part of the speech signal to obtain the IMFCC coefficients and fuse the MFCC features in order to obtain I_MFCC. Finally, to more accurately characterize emotional traits, we combined the Teager energy operator coefficients (TEOC) and the I_MFCC to obtain TEOC&I_MFCC and input it into the CNN_LSTM neural network. Experimental results on RAVDESS show that the feature fusion using Teager energy operator coefficients and I_MFCC has a higher emotion recognition accuracy, and the system achieves 92.99% weighted accuracy (WA) and 92.88% unweighted accuracy (UA).

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals.

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

Autistic adults exhibit highly precise representations of others’ emotions but a reduced influence of emotion representations on emotion recognition accuracy

Autistic adults exhibit highly precise representations of others’ emotions but a reduced influence of emotion representations on emotion recognition accuracy

Novel Emotion Recognition System Using Edge Computing Platform with Deep Convolutional Networks

Human faces have been naturally viewed as a central part in each image. One interesting task is to classify each face into different categories based on the emotion shown in the facial expression. In addition, an awareness of emotion during work on a project and how affective states are presented in the communication style might help system developers work more effectively, thus improving the performance of a collaborative team. Currently, the feasibility and portability of emotion recognition in the platform with Raspberry PI are insufficient. Hereby, a novel emotion recognition system in real time using the edge computing platform with deep learning has been implemented successfully. The feature values of objects are calculated by a high computing processor on the embedded platform. When an object with the matching features is detected, it is drawn as a rectangular bounding box and the results are displayed on the screen. In the proposed system, it first annotates the image datasets and saves them in the corresponding input data format for model training. Thus, the You Only Look Once (YOLOv5) model has been employed for training because it is a state-of-the-art object detection system. In other words, a fast and accurate emotion recognition is the main benefits of choosing YOLOv5 model. Then, the correctly trained YOLOv5 model file is loaded into an edge computing platform; and the feature values of objects are analyzed by a high computing processor. Finally, the experimental results show that the promising mean Average Precision (mAP), 92.6%, and recognition speed in Frames Per Second (FPS), 40, are obtained, which outperforms other existing systems.

AETIOLOGY ON EMOTION RECOGNITION STRATEGIES USING VIDEO, ELECTROENCEPHALOGRAM AND PHYSIOLOGICAL SIGNALS

Multimodal emotion recognition is a research area that involves using signals of various natures, such as facial expressions, speech, gestures, and physiological signals, to recognize emotions accurately. This field has gained importance due to its potential applications in decision-making, human recognition, and social interaction. One of the significant challenges in multimodal emotion recognition is feature extraction, which involves identifying the relevant features from the signals that carry emotional information. Various techniques have been developed for feature extraction, including machine learning-based methods, such as deep learning, feature fusion, and feature selection. These techniques aim to extract the most relevant and discriminative features from the signals to improve the accuracy of emotion recognition.

Autistic adults exhibit highly precise representations of others’ emotions but a reduced influence of emotion representations on emotion recognition accuracy

To date, studies have not yet established the mechanisms underpinning differences in autistic and non-autistic emotion recognition. The current study first investigated whether autistic and non-autistic adults differed in terms of the precision and/or differentiation of their visual emotion representations and their general matching abilities, and second, explored whether differences therein were related to challenges in accurately recognizing emotional expressions. To fulfil these aims, 45 autistic and 45 non-autistic individuals completed three tasks employing dynamic point light displays of emotional facial expressions. We identified that autistic individuals had more precise visual emotion representations than their non-autistic counterparts, however, this did not confer any benefit for their emotion recognition. Whilst for non-autistic people, non-verbal reasoning and the interaction between precision of emotion representations and matching ability predicted emotion recognition, no variables contributed to autistic emotion recognition. These findings raise the possibility that autistic individuals are less guided by their emotion representations, thus lending support to Bayesian accounts of autism.

Social cognitive deficit is associated with visuomotor coordination impairment and dopamine transporter availability in euthymic bipolar disorder

BackgroundExtensive evidence has suggested functional connections between co-occurring visuomotor and social cognitive deficits in neuropsychiatric disorders; however, such association has not been studied in bipolar disorder (BD). We aimed to investigate the relationship between visuomotor coordination and social cognition in the euthymic stage of BD (euBD). Given the shared neurobiological underpinnings involving the dopaminergic system and corticostriatal circuitry, we hypothesized a positive correlation between social cognition and visuomotor coordination in euBD patients. Methods40 euBD patients and 59 healthy control (HC) participants underwent evaluation of social (Diagnostic Analysis of Nonverbal Accuracy 2-Taiwan version (DANVA-2-TW)), non-social cognitive function and visuomotor coordination. A subgroup of participants completed single-photon emission computed tomography for striatal dopamine transporter (DAT) availability assessment. ResultsEuBD patients showed impaired nonverbal emotion recognition (ps ≤ 0.033) and poorer visuomotor coordination (ps < 0.003) compared to HC, with a positive correlation between these two abilities (r = 0.55, p < 0.01). However, after considering potential confounding factors, instead of visuomotor coordination, striatal DAT availability was a unique predictor of emotion recognition accuracy in euBD (beta = 0.33, p = 0.001). ConclusionOur study result supported a functional association between social cognition and visuomotor coordination in euBD, with striatal dopaminergic dysfunction emerged as a crucial contributing factor in their interrelation.

Trainee psychotherapists' emotion recognition accuracy improves after training: emotion recognition training as a tool for psychotherapy education.

Psychotherapists' emotional and empathic competencies have a positive influence on psychotherapy outcome and alliance. However, it is doubtful whether psychotherapy education in itself leads to improvements in trainee psychotherapists' emotion recognition accuracy (ERA), which is an essential part of these competencies. In a randomized, controlled, double-blind study (N = 68), we trained trainee psychotherapists (57% psychodynamic therapy and 43% cognitive behavioral therapy) to detect non-verbal emotional expressions in others using standardized computerized trainings - one for multimodal emotion recognition accuracy and one for micro expression recognition accuracy - and compared their results to an active control group one week after the training (n = 60) and at the one-year follow up (n = 55). The participants trained once weekly during a three-week period. As outcome measures, we used a multimodal emotion recognition accuracy task, a micro expression recognition accuracy task and an emotion recognition accuracy task for verbal and non-verbal (combined) emotional expressions in medical settings. The results of mixed multilevel analyses suggest that the multimodal emotion recognition accuracy training led to significantly steeper increases than the other two conditions from pretest to the posttest one week after the last training session. When comparing the pretest to follow-up differences in slopes, the superiority of the multimodal training group was still detectable in the unimodal audio modality and the unimodal video modality (in comparison to the control training group), but not when considering the multimodal audio-video modality or the total score of the multimodal emotion recognition accuracy measure. The micro expression training group showed a significantly steeper change trajectory from pretest to posttest compared to the control training group, but not compared to the multimodal training group. However, the effect vanished again until the one-year follow-up. There were no differences in change trajectories for the outcome measure about emotion recognition accuracy in medical settings. We conclude that trainee psychotherapists' emotion recognition accuracy can be effectively trained, especially multimodal emotion recognition accuracy, and suggest that the changes in unimodal emotion recognition accuracy (audio-only and video-only) are long-lasting. Implications of these findings for the psychotherapy education are discussed.

Emotion Recognition in Individuals with Down Syndrome: A Convolutional Neural Network-Based Algorithm Proposal

This research introduces an algorithm that automatically detects five primary emotions in individuals with Down syndrome: happiness, anger, sadness, surprise, and neutrality. The study was conducted in a specialized institution dedicated to caring for individuals with Down syndrome, which allowed for collecting samples in uncontrolled environments and capturing spontaneous emotions. Collecting samples through facial images strictly followed a protocol approved by certified Ethics Committees in Ecuador and Colombia. The proposed system consists of three convolutional neural networks (CNNs). The first network analyzes facial microexpressions by assessing the intensity of action units associated with each emotion. The second network utilizes transfer learning based on the mini-Xception architecture, using the Dataset-DS, comprising images collected from individuals with Down syndrome as the validation dataset. Finally, these two networks are combined in a CNN network to enhance accuracy. The final CNN processes the information, resulting in an accuracy of 85.30% in emotion recognition. In addition, the algorithm was optimized by tuning specific hyperparameters of the network, leading to a 91.48% accuracy in emotion recognition accuracy, specifically for people with Down syndrome.

Graph adaptive semi-supervised discriminative subspace learning for EEG emotion recognition

Since Electroencephalogram (EEG) is resistant to camouflage and contains abundant neurophysiological information, it shows significant superiorities in objective emotion recognition, making EEG-based emotion recognition become a hot research field in brain-computer interface research. However, EEG is generally non-stationary and has a low signal-to-noise ratio, which is difficult to analyze. Inspired by the consensus that exploring a discriminative subspace representation usually helps to capture the semantic information of EEG data, in this paper we propose a Graph Adaptive Semi-supervised Discriminative Subspace Learning (GASDSL) model for EEG-based emotion recognition. GASDSL aims to explore a discriminative subspace in which the intra-class scatter decreases while the inter-class separability increases. The adaptive maximum entropy graph construction and semi-supervised subspace emotional state prediction are adopted to mediate the discriminative subspace learning. Extensive comparative studies on the SEED-IV and SEED-V datasets depict that 1) GASDSL achieved satisfactory emotion recognition accuracy compared with other semi-supervised learning models, 2) the discriminative abilities of both the learned maximum entropy graph and subspace are improved as the model iterates, and 3) the features extracted from the Gamma band, the left/right temporal, prefrontal, and (central) parietal lobes contributed more to emotion recognition based on the spatial-frequency pattern analysis results.

A deep learning-based approach for emotional analysis of sports dance.

There is a phenomenon of attaching importance to technique and neglecting emotion in the training of sports dance (SP), which leads to the lack of integration between movement and emotion and seriously affects the training effect. Therefore, this article uses the Kinect 3D sensor to collect the video information of SP performers and obtains the pose estimation of SP performers by extracting the key feature points. The Arousal-Valence (AV) emotion model, based on the Fusion Neural Network model (FUSNN), is also combined with theoretical knowledge. It replaces long short term memory (LSTM) with gate recurrent unit (GRU), adds layer-normalization and layer-dropout, and reduces stack levels, and it is used to categorize SP performers' emotions. The experimental results show that the model proposed in this article can accurately detect the key points in the performance of SP performers' technical movements and has a high emotional recognition accuracy in the tasks of 4 categories and eight categories, reaching 72.3% and 47.8%, respectively. This study accurately detected the key points of SP performers in the presentation of technical movements and made a major contribution to the emotional recognition and relief of this group in the training process.

FCAN–XGBoost: A Novel Hybrid Model for EEG Emotion Recognition

In recent years, artificial intelligence (AI) technology has promoted the development of electroencephalogram (EEG) emotion recognition. However, existing methods often overlook the computational cost of EEG emotion recognition, and there is still room for improvement in the accuracy of EEG emotion recognition. In this study, we propose a novel EEG emotion recognition algorithm called FCAN–XGBoost, which is a fusion of two algorithms, FCAN and XGBoost. The FCAN module is a feature attention network (FANet) that we have proposed for the first time, which processes the differential entropy (DE) and power spectral density (PSD) features extracted from the four frequency bands of the EEG signal and performs feature fusion and deep feature extraction. Finally, the deep features are fed into the eXtreme Gradient Boosting (XGBoost) algorithm to classify the four emotions. We evaluated the proposed method on the DEAP and DREAMER datasets and achieved a four-category emotion recognition accuracy of 95.26% and 94.05%, respectively. Additionally, our proposed method reduces the computational cost of EEG emotion recognition by at least 75.45% for computation time and 67.51% for memory occupation. The performance of FCAN–XGBoost outperforms the state-of-the-art four-category model and reduces computational costs without losing classification performance compared with other models.

Facial emotion recognition using deep learning detector and classifier

Numerous research works have been put forward over the years to advance the field of facial expression recognition which until today, is still considered a challenging task. The selection of image color space and the use of facial alignment as preprocessing steps may collectively pose a significant impact on the accuracy and computational cost of facial emotion recognition, which is crucial to optimize the speed-accuracy trade-off. This paper proposed a deep learning-based facial emotion recognition pipeline that can be used to predict the emotion of detected face regions in video sequences. Five well-known state-of-the-art convolutional neural network architectures are used for training the emotion classifier to identify the network architecture which gives the best speed-accuracy trade-off. Two distinct facial emotion training datasets are prepared to investigate the effect of image color space and facial alignment on the performance of facial emotion recognition. Experimental results show that training a facial expression recognition model with grayscale-aligned facial images is preferable as it offers better recognition rates with lower detection latency. The lightweight MobileNet_v1 is identified as the best-performing model with WM=0.75 and RM=160 as its hyper-parameters, achieving an overall accuracy of 86.42% on the testing video dataset.