Emotion Recognition Performance Research Articles

Emotion Recognition Through Combining EEG and EOG Over Relevant Channels With Optimal Windowing

For dimensional emotion recognition, electroencephalography (EEG) signals and electrooculogram (EOG) signals are often combined to improve the performance of classifiers, as each of them provides complementary features to the other. In this article, we combine the EEG signal on the relevant channels with the EOG signal to boost the recognition accuracy. We first explore the mutual information (MI) of all EEG channels and only select emotion-related channels, i.e., channels with more MI are retained, since the emotion recognition performance can be degraded by the interference between uncorrelated channels, while the computational complexity is significant if all EEG channels are used for recognition. While the optimal lengths of EEG and EOG signals for emotion recognition are still uncertain, we systematically investigate the effects of time-window size on emotion recognition. This strategy not only increases the number of training samples, but also reduces the feature redundancy. At this stage, we not only extract multiple statistical features but also employ the increment entropy to find abrupt changes in EEG signals. The experimental results show that 13 out of 32 EEG channels were selected by the proposed channel selection algorithm, and these selected channels can already produce accurate emotion predictions. We found that using optimal time-windows to split EEG and EOG signals into several thin slices and then combine them can further enhance the emotion recognition performance, where the time-windows of 4, 5, 6, and 10 s allow the combined signals to achieve very high accuracy.

Developing a Physiological Signal-Based, Mean Threshold and Decision-Level Fusion Algorithm (PMD) for Emotion Recognition

With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.

Spatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition

Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.

Multi-channel EEG emotion recognition through residual graph attention neural network.

In this paper, a novel EEG emotion recognition method based on residual graph attention neural network is proposed. The method constructs a three-dimensional sparse feature matrix according to the relative position of electrode channels, and inputs it into the residual network to extract high-level abstract features containing electrode spatial position information. At the same time, the adjacency matrix representing the connection relationship of electrode channels is constructed, and the time-domain features of multi-channel EEG are modeled using graph. Then, the graph attention neural network is utilized to learn the intrinsic connection relationship between EEG channels located in different brain regions from the adjacency matrix and the constructed graph structure data. Finally, the high-level abstract features extracted from the two networks are fused to judge the emotional state. The experiment is carried out on DEAP data set. The experimental results show that the spatial domain information of electrode channels and the intrinsic connection relationship between different channels contain salient information related to emotional state, and the proposed model can effectively fuse these information to improve the performance of multi-channel EEG emotion recognition.

Cross Corpus Speech Emotion Recognition using transfer learning and attention-based fusion of Wav2Vec2 and prosody features

Speech Emotion Recognition (SER) performance degrades when their training and test conditions or corpora differ. Cross-corpus SER (CCSER) is a research branch that discusses adapting an SER system to identify speech emotions on a corpus that has different recording conditions or language from the training corpus. For CCSER, adaption can be performed in the feature extraction module or emotion classifier, which are the two main components of the SER system. In this paper, we propose AFTL method (attention-based feature fusion along with transfer learning), including methods in both feature extraction and classification for CCSER. In the feature extraction part, we use Wav2Vec 2.0 transformer blocks and prosody features, and we propose an attention method for fusing them. In the classifier part, we use transfer learning for transferring the knowledge of a model trained on source emotional speech corpus to recognize emotions on a target corpus. We performed experiments on numerous speech emotional datasets as target corpora, where we used IEMOCAP as the source corpus. For instance, we achieve 92.45% accuracy on the EmoDB dataset, where we only use 20% of speakers for adapting the source model. In addition, for other target corpora, we obtained admissible results.

Emotion recognition and mood along the menstrual cycle

Previous studies have demonstrated menstrual cycle dependent changes in the recognition of facial emotional expressions, specifically the expression of fear, anger, sadness or disgust. While some studies demonstrate an improvement of emotion recognition performance during the peri-ovulatory phase, when estradiol levels peak, other studies demonstrate a deterioration of emotion recognition performance during the mid-luteal phase, when progesterone levels peak. It has been hypothesized, that these changes in emotion recognition performance mirror mood changes along the menstrual cycle. In the present study, we investigate, whether changes in emotion recognition performance along the menstrual cycle are mediated by mood changes along the menstrual cycle. In a combined cross-sectional and longitudinal study design, two large samples of women completed an emotion recognition task, as well as several mood questionnaires during their menses, peri-ovulatory or mid-luteal cycle phase. 65 women completed the task thrice, once during each cycle phase, order counterbalanced. In order to control for potential learning effects, a sample of 153 women completed the task only once in one of the three cycle phases. In both samples, results demonstrated no significant changes in emotion recognition performance along the menstrual cycle, irrespective of the performance measure investigated (accuracy, reaction time, frequency of emotion classifications) and irrespective of the emotion displayed. Bayesian statistics provided very strong evidence for the null hypothesis, that emotion recognition does not change along the menstrual cycle. There was also no moderation of emotion recognition changes along the menstrual cycle by mood changes along the menstrual cycle. Mood changes along the menstrual cycle followed the expected pattern with highest positive affect and least premenstrual symptoms around ovulation and lowest positive affect, but strongest premenstrual symptoms during menses. Interestingly, premenstrual symptoms were negatively related to estradiol, suggesting a protective effect of estrogen during the luteal cycle phase against mood worsening during the premenstrual phase.

Trusted emotion recognition based on multiple signals captured from video

Emotion recognition based on facial expressions has low accuracy and doubtful reliability because of the existence of fake expressions. In this paper, a method is proposed to recognize fake emotions based on multivisual information generated from single information sources, including facial expressions, eye states and physiological signals captured from video. An algorithm based on a graph neural network is used to extract spatial and spectral domain features from facial images for facial expression recognition. A model-based method is used for decomposing RGB signals into heart rates. A deep model trained by a labeled dataset that we created is used to segment the eye region. After obtaining the signals extracted from video, different fusion strategies are applied to evaluate emotion recognition performance based on multiple signals. In the experiment, the CK+, TFEID, JAFFE, RAF, PURE, and ESLD datasets are used to measure the accuracy of facial expression recognition, heart rate detection and eye region segmentation. The results show that multimodality is effective in improving accuracy and that eye state can be considered a cue for trusted emotion recognition. Compared with the method based on the eye state and noncontact physiological signal, the accuracy of multimodality can be improved by 26.19% and 9.52%, respectively.

Mediating effects of impulsivity and alexithymia in the association between traumatic brain injury and aggression in incarcerated males.

Studies suggest both alexithymia and impulsivity (partially) explain aggressive behavior in traumatic brain injury (TBI) patients, but none of these studies use both questionnaire and performance-based measures as recommended, nor simultaneously investigate both impulsivity and alexithymia. The available studies therefore likely miss part of the constructs of alexithymia and impulsivity, and do not comprehensively assess the mediating effects of both constructs in the relationship between TBI and aggression. A sample of N = 281 incarcerated individuals were recruited from Dutch penitentiary institutions, and completed the Buss Perry Aggression Questionnaire (aggression), BIS-11 (impulsivity) and Toronto Alexithymia Scale-20 (alexithymia) questionnaires, as well as a stop-signal task and an emotion recognition paradigm. Several multiple mediation analyses were conducted using structural equation modelling, to assess the viability of a causal theoretical model of aggression. The final planned models were the original models with a good fit with the data (comparative fit index > 0.95, root mean square error of approximation and Standardized root mean square residual < 0.05), and results indicate that only questionnaire-based impulsivity mediated the relationship between TBI and aggression. TBI was unrelated to alexithymia, stop-signal or emotion recognition performance. Aggression was predicted by both alexithymia and impulsivity, but not by the performance measures. Post hoc analyses shows that alexithymia moderates the relationship between impulsivity and aggression. These results imply that aggressive incarcerated individuals showing impulsive behavior should be screened for TBI, since TBI is often overlooked or misdiagnosed, and indicate that both impulsivity and alexithymia are potential focus points for aggression reduction treatment in TBI patients.

Joint EEG Feature Transfer and Semisupervised Cross-Subject Emotion Recognition

Due to the weak and non-stationary properties, Electroencephalogram (EEG) data presents significant individual differences. To align data distributions of different subjects, transfer learning showed promising performance in cross-subject EEG emotion recognition. However, most of the existing models sequentially learned the domain-invariant features and estimated the target domain label information. Such a two-stage strategy breaks the inner connections of both processes, inevitably causing the sub-optimality. In this paper, we propose a Joint EEG feature Transfer and Semi-supervised cross-subject emotion Recognition (JTSR) model in which the shared subspace projection matrix and target label are jointly optimized towards the optimum. Extensive experiments are conducted on SEED-IV and SEED, and the results show that 1) the emotion recognition performance is significantly enhanced by the joint learning mode, and 2) the spatial-frequency activation patterns of critical EEG frequency bands and brain regions in cross-subject emotion expression are quantitatively identified by analyzing the learned shared subspace.

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

EEG-Based Emotion Recognition With Emotion Localization via Hierarchical Self-Attention

Emotion recognition based on electroencephalography (EEG) has attracted significant attention due to its wide range of applications, especially in Human-Computer Interaction(HCI). Previous research treats different segments of EEG signals uniformly, ignoring the fact that emotions are unstable and discrete during an extended period. In this paper, we propose a novel two-step spatial-temporal emotion recognition framework. First, considering that the human emotion has not only "short-term continuity" but also "long-term similarity", we propose a hierarchical self-attention network to jointly model local and global temporal information, so as to localize most related segments and reduce the influence of noise at the temporal level. Second, in order to extract discriminative features at the spatial level to enhance the emotion recognition performance, we further employ the squeeze-and-excitation module (SE module) along with the channel correlation loss (CC-Loss) to select the most task-related channels. We also define a new task called <i>emotion localization</i>, which aims to localize fragments with stronger emotions. We evaluate the proposed method on the proposed emotion localization task and typical emotion recognition task with three publicly available datasets, i.e., SEED, DEAP, and MAHNOB-HCI. The experimental results demonstrate that the proposed approach outperforms state-of-the-art methods.

Multimodal Prompt Learning in Emotion Recognition Using Context and Audio Information

Prompt learning has improved the performance of language models by reducing the gap in language model training methods of pre-training and downstream tasks. However, extending prompt learning in language models pre-trained with unimodal data to multimodal sources is difficult as it requires additional deep-learning layers that cannot be attached. In the natural-language emotion-recognition task, improved emotional classification can be expected when using audio and text to train a model rather than only natural-language text. Audio information, such as voice pitch, tone, and intonation, can give more information that is unavailable in text to predict emotions more effectively. Thus, using both audio and text can enable better emotion prediction in speech emotion-recognition models compared to semantic information alone. In this paper, in contrast to existing studies that use multimodal data with an additional layer, we propose a method for improving the performance of speech emotion recognition using multimodal prompt learning with text-based pre-trained models. The proposed method is using text and audio information in prompt learning by employing a language model pre-trained on natural-language text. In addition, we propose a method to improve the emotion-recognition performance of the current utterance using the emotion and contextual information of the previous utterances for prompt learning in speech emotion-recognition tasks. The performance of the proposed method was evaluated using the English multimodal dataset MELD and the Korean multimodal dataset KEMDy20. Experiments using both the proposed methods obtained an accuracy of 87.49%, F1 score of 44.16, and weighted F1 score of 86.28.

Knowledge-based BERT word embedding fine-tuning for emotion recognition

Emotion recognition has received considerable attention in recent years, with the popularity of social media. It is noted, however, that the state-of-the-art language models such as Bidirectional Encoder Representations from Transformers (BERT) may not produce the best performance in emotion recognition. We found the main cause of the problem is that the embedding of emotional words from the pre-trained BERT model may not exhibit high between-class difference and within-class similarity. While BERT model fine-tuning is a common practice when it is applied to specific tasks, this may not be practical in emotion recognition because most datasets are small and some texts are short and noisy, without containing much useful contextual information. In this paper, we propose to use the knowledge of emotion vocabulary to fine-tune embedding of emotional words. As a separate module independent of the embedding learning model, the fine-tuning model aims to produce emotional word embedding with improved within-class similarity and between-class difference. By combining the emotionally discriminative fine-tuned embedding with contextual information-rich embedding from pre-trained BERT model, the emotional features underlying the texts could be more effectively captured in the subsequent feature learning module, which in turn leads to improved emotion recognition performance. The knowledge-based word embedding fine-tuning model is tested on five datasets of emotion recognition, and the results and analysis demonstrate the effectiveness of the proposed method.

Facial emotion recognition using deep learning detector and classifier

Numerous research works have been put forward over the years to advance the field of facial expression recognition which until today, is still considered a challenging task. The selection of image color space and the use of facial alignment as preprocessing steps may collectively pose a significant impact on the accuracy and computational cost of facial emotion recognition, which is crucial to optimize the speed-accuracy trade-off. This paper proposed a deep learning-based facial emotion recognition pipeline that can be used to predict the emotion of detected face regions in video sequences. Five well-known state-of-the-art convolutional neural network architectures are used for training the emotion classifier to identify the network architecture which gives the best speed-accuracy trade-off. Two distinct facial emotion training datasets are prepared to investigate the effect of image color space and facial alignment on the performance of facial emotion recognition. Experimental results show that training a facial expression recognition model with grayscale-aligned facial images is preferable as it offers better recognition rates with lower detection latency. The lightweight MobileNet_v1 is identified as the best-performing model with WM=0.75 and RM=160 as its hyper-parameters, achieving an overall accuracy of 86.42% on the testing video dataset.

STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition.

How to use the characteristics of EEG signals to obtain more complementary and discriminative data representation is an issue in EEG-based emotion recognition. Many studies have tried spatio-temporal or spatio-spectral feature fusion to obtain higher-level representations of EEG data. However, these studies ignored the complementarity between spatial, temporal and spectral domains of EEG signals, thus limiting the classification ability of models. This study proposed an end-to-end network based on ManifoldNet and BiLSTM networks, named STSNet. The STSNet first constructed a 4-D spatio-temporal-spectral data representation and a spatio-temporal data representation based on EEG signals in manifold space. After that, they were fed into the ManifoldNet network and the BiLSTM network respectively to calculate higher-level features and achieve spatio-temporal-spectral feature fusion. Finally, extensive comparative experiments were performed on two public datasets, DEAP and DREAMER, using the subject-independent leave-one-subject-out cross-validation strategy. On the DEAP dataset, the average accuracy of the valence and arousal are 69.38% and 71.88%, respectively; on the DREAMER dataset, the average accuracy of the valence and arousal are 78.26% and 82.37%, respectively. Experimental results show that the STSNet model has good emotion recognition performance.

Performance in emotion recognition and theory of mind tasks in social anxiety and generalized anxiety disorders: a systematic review and meta-analysis.

Social cognition impairments may be associated with poor functional outcomes, symptoms, and disability in social anxiety disorder (SAD) and generalized anxiety disorder (GAD). This meta-analysis aims to determine if emotion recognition and theory of mind (ToM) are impaired in SAD or GAD compared to healthy controls. A systematic review was conducted in electronic databases (PubMed, PsycNet, and Web of Science) to retrieve studies assessing emotion recognition and/or ToM in patients with SAD or GAD, compared to healthy controls, up to March 2022. Meta-analyses using random-effects models were conducted. We identified 21 eligible studies: 13 reported emotion recognition and 10 ToM outcomes, with 585 SAD patients, 178 GAD patients, and 753 controls. Compared to controls, patients with SAD exhibited impairments in emotion recognition (SMD = -0.32, CI = -0.47 - -0.16, z = -3.97, p < 0.0001) and ToM (SMD = -0.44, CI = -0.83 -0.04, z = -2.18, p < 0.01). Results for GAD were inconclusive due to the limited number of studies meeting the inclusion criteria (two for each domain). Relevant demographic and clinical variables (age, sex, education level, and anxiety scores) were not significantly correlated with emotion recognition or ToM impairments in SAD and GAD. Further studies employing ecological measures with larger and homogenous samples are needed to better delineate the factors influencing social cognition outcomes in both SAD and GAD.

Multi-modal speech emotion detection using optimised deep neural network classifier

ABSTRACT Speech emotion recognition (SER) has received significant attention recently and seems to be a critical aspect of human-computer interaction. The performance of the recent methods is not at the expected level there are many different strategies have been developed for SER. In the present situation, communication, non-verbal vocalisation, with a vocal sound plays an essential role in emotional expression. The multimodal database is considered for which the hybridised model of the audio-visual dependent emotion identification model is proposed. In this research, the optimised deep NN is used to recognise emotions from multimodal input data. For better performance in emotion recognition, the learner memorising optimisation fine-tunes the hyperparameters involved in the deep NN network classifier. The hybrid texture feature descriptor is proposed to improve classification outcomes from audio-video signals as well as accuracy. The learner memorising optimisation is used in the deep NN classifier to obtain the ideal parameters involved in exploring space with high accuracy. The metric values of the learner memorising optimisation-based deep NN have been evaluated using the Enter face’05 database and the proposed method obtained 97.490% accuracy, 98% sensitivity, and 97.490% specificity for the K-fold value 10% and 96.928%, 98.80%, and 96.928% for 90% training data.

Ocular emotion discrimination disorders in self-limited epilepsy patients with centrotemporal spikes complicated with electrical status epilepticus during sleep: A pediatric neuropsychological study.

We investigated the characteristics and factors influencing eye emotion recognition in self-limited epilepsy patients with centrotemporal spikes (SeLECTS) complicated with electrical status epilepticus during sleep (ESES). We selected SeLECTS (n=160) patients treated in the outpatient and inpatient departments of Anhui Children's Hospital from September 2020 to January 2022. According to the video electroencephalogram monitoring slow-wave index (SWI), SeLECTS patients with SWI<50% were assigned into the typical SeLECTS group (n=79), and patients with SWI ≥ 50% were assigned into the ESES group (n=81). Patients in the two groups were assessed by The Eye Basic Emotion Discrimination Task (EBEDT) and The Eye Complex Emotion Discrimination Task (ECEDT), respectively. Comparisons were made with age-, sex- and education level-matched healthy control participants. The correlation between the characteristics of emotional discrimination disorder in the eye area and the clinical influencing factors was analyzed in ESES group, and p ≤ .050 was the threshold for significance. Relative to the healthy control group, scores of sadness and fear in the typical SeLECTS group were markedly lower (p=.018, p=.023), while differences in scores of disgust, happiness, surprise, and anger were not significantly different between the groups (p=.072, p=.162, p=.395, p=.380, respectively). Compared with the healthy control group, the ESES group had significantly low scores in recognition of sadness, fear, disgust, and surprise (p=.006, p=.016, p=.043 and p=.038, respectively). However, differences in recognition of happiness and anger between the groups were not significant (p=.665 and p=.272). Univariate logistic analysis showed that the score of eye recognition for sadness in the ESES group was affected by age of onset, SWI, ESES duration and number of seizures. The score of eye recognition for fear was mainly affected by SWI, while the score of eye recognition for disgust was affected by SWI and number of seizures. The surprised eye emotion recognition score was mainly affected by the number of seizures. Variables with p<.1 were considered to be independent variables of multivariable ordered logistic regression. Multivariate logistic analysis showed that sadness emotion recognition was mainly affected by SWI and ESES duration, while disgust was mainly affected by SWI. The typical SeLECTS group showed impaired emotion (sadness and fear) recognition function in the eye area. The ESES group was associated with more intense emotional (sadness, fear, disgust, and surprise) recognition impairment in the eye region. The higher the SWI, the younger the onset age and the longer the duration of ESES, while the more the number of seizures, the more serious the impairment of emotional recognition function in the affected eye area.

Structural integrity of the insula and emotional facial recognition performance following stroke.

The role of the human insula in facial emotion recognition is controversially discussed, especially in relation to lesion-location-dependent impairment following stroke. In addition, structural connectivity quantification of important white-matter tracts that link the insula to impairments in facial emotion recognition has not been investigated. In a case-control study, we investigated a group of 29 stroke patients in the chronic stage and 14 healthy age- and gender-matched controls. Lesion location of stroke patients was analysed with voxel-based lesion-symptom mapping. In addition, structural white-matter integrity for tracts between insula regions and their primarily known interconnected brain structures was quantified by tractography-based fractional anisotropy. Our behavioural analyses showed that stroke patients were impaired in the recognition of fearful, angry and happy but not disgusted expressions. Voxel-based lesion mapping revealed that especially lesions centred around the left anterior insula were associated with impaired recognition of emotional facial expressions. The structural integrity of insular white-matter connectivity was decreased for the left hemisphere and impaired recognition accuracy for angry and fearful expressions was associated with specific left-sided insular tracts. Taken together, these findings suggest that a multimodal investigation of structural alterations has the potential to deepen our understanding of emotion recognition impairments after stroke.

Effects of mask use and other‐race on face perception, emotion recognition, and social distancing during the COVID‐19 pandemic

AbstractWe tested the effect of mask use and other‐race effect on (a) face recognition, (b) recognition of facial expressions, and (c) social distance. Caucasian subjects were tested in a matching‐to‐sample paradigm with either masked or unmasked Caucasian and Asian faces. The participants exhibited the best performance in recognizing an unmasked face condition and the poorest to recognize a masked face that they had seen earlier without a mask. Accuracy was poorer for Asian faces than Caucasian faces. The second experiment presented Asian or Caucasian faces having emotional expressions, with and without masks. The participants' emotion recognition performance decreased for masked faces. From the most accurately to least accurately recognized emotions were as follows: happy, neutral, disgusted, fearful. Performance was poorer for Asian stimuli compared to Caucasian. In Experiment 3 the same participants indicated the social distance they would prefer with each pictured person. They preferred a wider distance with unmasked faces compared to masked faces. Distance from farther to closer was as follows: disgusted, fearful, neutral, and happy. They preferred wider social distance for Asian compared to Caucasian faces. Altogether, findings indicated that during the COVID‐19 pandemic mask wearing decreased recognition of faces and emotional expressions, negatively impacting communication among people from different ethnicities.