General Emotion Recognition Research Articles

S[formula omitted]T-Net: A novel electroencephalogram signals-oriented emotion recognition model

In this paper, a novel skipping spatial–spectral–temporal network (S3T-Net) is developed to handle intra-individual differences in electroencephalogram (EEG) signals for accurate, robust, and generalized emotion recognition. In particular, aiming at the 4D features extracted from the raw EEG signals, a multi-branch architecture is proposed to learn spatial–spectral cross-domain representations, which benefits enhancing the model generalization ability. Time dependency among different spatial–spectral features is further captured via a bi-directional long-short term memory module, which employs an attention mechanism to integrate context information. Moreover, a skip-change unit is designed to add another auxiliary pathway for updating model parameters, which alleviates the vanishing gradient problem in complex spatial–temporal network. Evaluation results show that the proposed S3T-Net outperforms other advanced models in terms of the emotion recognition accuracy, which yields an performance improvement of 0.23% , 0.13%, and 0.43% as compared to the sub-optimal model in three test scenes, respectively. In addition, the effectiveness and superiority of the key components of S3T-Net are demonstrated from various experiments. As a reliable and competent emotion recognition model, the proposed S3T-Net contributes to the development of intelligent sentiment analysis in human–computer interaction (HCI) realm.

GPT-4V with emotion: A zero-shot benchmark for Generalized Emotion Recognition

Recently, GPT-4 with Vision (GPT-4V) has demonstrated remarkable visual capabilities across various tasks, but its performance in emotion recognition has not been fully evaluated. To bridge this gap, we present the quantitative evaluation results of GPT-4V on 21 benchmark datasets covering 6 tasks: visual sentiment analysis, tweet sentiment analysis, micro-expression recognition, facial emotion recognition, dynamic facial emotion recognition, and multimodal emotion recognition. This paper collectively refers to these tasks as “Generalized Emotion Recognition (GER)”. Through experimental analysis, we observe that GPT-4V exhibits strong visual understanding capabilities in GER tasks. Meanwhile, GPT-4V shows the ability to integrate multimodal clues and exploit temporal information, which is also critical for emotion recognition. However, it is worth noting that GPT-4V is primarily designed for general domains and cannot recognize micro-expressions that require specialized knowledge. To the best of our knowledge, this paper provides the first quantitative assessment of GPT-4V for GER tasks. We have open-sourced the code and encourage subsequent researchers to broaden the evaluation scope by including more tasks and datasets. Our code and evaluation results are available at: https://github.com/zeroQiaoba/gpt4v-emotion.

TSception: Capturing Temporal Dynamics and Spatial Asymmetry From EEG for Emotion Recognition

The high temporal resolution and the asymmetric spatial activations are essential attributes of electroencephalogram (EEG) underlying emotional processes in the brain. To learn the temporal dynamics and spatial asymmetry of EEG towards accurate and generalized emotion recognition, we propose TSception, a multi-scale convolutional neural network that can classify emotions from EEG. TSception consists of dynamic temporal, asymmetric spatial, and high-level fusion layers, which learn discriminative representations in the time and channel dimensions simultaneously. The dynamic temporal layer consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of EEG, which learns the dynamic temporal and frequency representations of EEG. The asymmetric spatial layer takes advantage of the asymmetric EEG patterns for emotion, learning the discriminative global and hemisphere representations. The learned spatial representations will be fused by a high-level fusion layer. Using more generalized cross-validation settings, the proposed method is evaluated on two publicly available datasets DEAP and MAHNOB-HCI. The performance of the proposed network is compared with prior reported methods such as SVM, KNN, FBFgMDM, FBTSC, Unsupervised learning, DeepConvNet, ShallowConvNet, and EEGNet. TSception achieves higher classification accuracies and F1 scores than other methods in most of the experiments. The codes are available at https://github.com/yi-ding-cs/TSception

Can third-party observers detect attraction in others based on subtle nonverbal cues?

In a series of three studies, we examined whether third-party observers can detect attraction in others based on subtle nonverbal cues. We employed video segments of dates collected from a speed-dating experiment, in which daters went on a brief (approx. 4 min) blind-date and indicated whether they would like to go on another date with their brief interaction partner or not. We asked participants to view these stimuli and indicate whether or not each couple member is attracted to their partner. Our results show that participants could not reliably detect attraction, and this ability was not influenced by the age of the observer, video segment location (beginning or middle of the date), video duration, or general emotion recognition capacity. Contrary to previous research findings, our findings suggest that third-party observers cannot reliably detect attraction in others. However, there was one exception: Recognition rose above chance level when the daters were both interested in their partners compared to when they were not interested.

RECOGNITION OF MENTAL DISORDERS FROM PHYSIOLOGICAL SIGNALS ANALYSIS

The rapid advances in machine learning (ML) and information fusion have made it possible to use machines/computers with the ability of understanding, recognition, and analysis of human emotion, mood and stress, and related mental diseases. The recognition methods based on physiological modalities are the most performant. Wearable technologies enable non-invasive long-term data gathering and analysis. The number of mental health issues are correlated with emotional states and can be possibly detected by similar methods to general emotion recognition. The scientific interest in the recognition of mental disorders is growing, and most of the available studies are uni-modal based on either ECG or EEG sensor data, while some recent studies also utilize multiple modalities and sensor fusion.

GameEmo-CapsNet: Emotion Recognition from Single-Channel EEG Signals Using the 1D Capsule Networks

Human emotion recognition with machine learning methods through electroencephalographic (EEG) signals has become a highly interesting subject for researchers. Although it is simple to define emotions that can be expressed physically such as speech, facial expressions, and gestures, it is more difficult to define psychological emotions that are expressed internally. The most important stimuli in revealing inner emotions are aural and visual stimuli. In this study, EEG signals using both aural and visual stimuli were examined and emotions were evaluated in both binary and multi-class emotion recognitions models. A general emotion recognition model was proposed for non-subject-based classification. Unlike in previous studies, a subject-based testing was performed for the first time in the literature. Capsule Networks, a new neural network model, has been developed for binary and multi-class emotion recognition. In the proposed method, a novel fusion strategy was introduced for binary-class emotion recognition and the model was tested using the GAMEEMO dataset. Binary-class emotion recognition achieved a classification accuracy which was 10% better than the classification performance achieved in other studies in the literature. Based on these findings, we suggest that the proposed method will bring a different perspective to emotion recognition.

A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals

Emotion recognition using brain signals has the potential to change the way we identify and treat some health conditions. Difficulties and limitations may arise in general emotion recognition software due to the restricted number of facial expression triggers, dissembling of emotions, or among people with alexithymia. Such triggers are identified by studying the continuous brainwaves generated by human brain. Electroencephalogram (EEG) signals from the brain give us a more diverse insight on emotional states that one may not be able to express. Brainwave EEG signals can reflect the changes in electrical potential resulting from communications networks between neurons. This research involves analyzing the epoch data from EEG sensor channels and performing comparative analysis of multiple machine learning techniques [namely Support Vector Machine (SVM), K-nearest neighbor, Linear Discriminant Analysis, Logistic Regression and Decision Trees each of these models] were tested with and without principal component analysis (PCA) for dimensionality reduction. Grid search was also utilized for hyper-parameter tuning for each of the tested machine learning models over Spark cluster for lowered execution time. The DEAP Dataset was used in this study, which is a multimodal dataset for the analysis of human affective states. The predictions were based on the labels given by the participants for each of the 40 1-min long excerpts of music. music. Participants rated each video in terms of the level of arousal, valence, like/dislike, dominance and familiarity. The binary class classifiers were trained on the time segmented, 15 s intervals of epoch data, individually for each of the 4 classes. PCA with SVM performed the best and produced an F1-score of 84.73% with 98.01% recall in the 30th to 45th interval of segmentation. For each of the time segments and “a binary training class” a different classification model converges to a better accuracy and recall than others. The results prove that different classification models must be used to identify different emotional states.

An Emotion Recognition-Awareness Vulnerability Hypothesis for Depression in Adolescence: A Systematic Review.

In this systematic review, we examined the evidence for an adolescent emotion recognition-awareness vulnerability to depression. The current review provided a qualitative synthesis of the emotion recognition (26 studies) and emotion awareness (38 studies) literatures for adolescent depression and was grounded within the framework of affective social competence (Halberstadt et al. in Soc Dev 10:79-119, 2001). It was hypothesized that deficits or difficulties in recognizing emotions in others and in being aware of emotions within the self would increase vulnerability to depression for adolescents. There was limited evidence to support a general emotion recognition vulnerability due to heterogeneous research designs and inconsistencies across studies; however, three emerging trends in specific recognition deficits associated with adolescent depression were identified: sensitivity to sadness, under-perceiving happiness, and over-perceiving anger. In contrast, there was robust support for an emotion awareness vulnerability to depression from cross-sectional, longitudinal, and longitudinal onset studies. When recognition and awareness are considered together, this review indicated that deficits in emotion awareness may exacerbate difficulties with emotion recognition during stressful or emotionally evocative contexts. This review highlighted the need for future investigations into emotion recognition and emotion awareness deficits in relation to adolescent depression using methodological innovations and longitudinal, clinical designs.

A Globally Generalized Emotion Recognition System Involving Different Physiological Signals.

Machine learning approaches for human emotion recognition have recently demonstrated high performance. However, only/mostly for subject-dependent approaches, in a variety of applications like advanced driver assisted systems, smart homes and medical environments. Therefore, now the focus is shifted more towards subject-independent approaches, which are more universal and where the emotion recognition system is trained using a specific group of subjects and then tested on totally new persons and thereby possibly while using other sensors of same physiological signals in order to recognize their emotions. In this paper, we explore a novel robust subject-independent human emotion recognition system, which consists of two major models. The first one is an automatic feature calibration model and the second one is a classification model based on Cellular Neural Networks (CNN). The proposed system produces state-of-the-art results with an accuracy rate between and when using the same elicitation materials and physiological sensors brands for both training and testing and an accuracy rate of when the elicitation materials and physiological sensors brands used in training are different from those used in training. Here, the following physiological signals are involved: ECG (Electrocardiogram), EDA (Electrodermal activity) and ST (Skin-Temperature).

Exploring emotion regulation and emotion recognition in people with presymptomatic Huntington's disease: The role of emotional awareness

Interest in the role of both emotion regulation and recognition in our understanding of mental health has been steadily increasing, especially in people with chronic illness who also have psychological difficulties. One illness which belongs to this category is Huntington's disease. Huntington's disease (HD) is a chronic neurodegenerative disorder that can cause a number of cognitive and psychological difficulties, including emotion recognition deficits, even before the onset of the symptoms required to make a formal diagnosis. Despite the lack of definite evidence, recent studies have suggested that deficits of emotion regulation and recognition may be expected to play a pivotal role in the early cognitive manifestations of HD.In this study, we hypothesised that the ability to regulate emotions can be impaired in people with presymptomatic HD, and that such impairment may be associated with a deficit of emotion recognition. To test this, an online survey was carried out with 117 English and Italian-speaking people with presymptomatic HD, compared to 217 controls matched for age and education.The results suggest that, in presymptomatic participants, emotion regulation and emotion recognition are generally not significantly impaired, and no significant relationships between performances on the two constructs were observed. However, a specific impairment in emotional awareness (a subscale on the Difficulties in Emotion Regulation Scale, DERS) was observed, which appears to be enhanced by the co-occurrence of depressive symptoms, even at a subclinical level. Consequently, it is suggested that difficulties in emotional awareness may represent a precursor of more general emotion recognition impairments, which only become apparent as the disease reaches a more symptomatic level. Clinical implications of the findings are discussed and directions for future research are proposed.

In your face: the biased judgement of fear-anger expressions in violent offenders

BackgroundWhy is it that certain violent criminals repeatedly find themselves engaged in brawls? Many inmates report having felt provoked or threatened by their victims, which might be due to a tendency to ascribe malicious intentions when faced with ambiguous social signals, termed hostile attribution bias.MethodsThe present study presented morphed fear-anger faces to prison inmates with a history of violent crimes, a history of child sexual abuse, and to matched controls form the general population. Participants performed a fear-anger decision task. Analyses compared both response frequencies and measures derived from psychophysical functions fitted to the data. In addition, a test to distinguish basic facial expressions and questionnaires for aggression, psychopathy and personality disorders were administered.ResultsViolent offenders present with a reliable hostile attribution bias, in that they rate ambiguous fear-anger expressions as more angry, compared to both the control population and perpetrators of child sexual abuse. Psychometric functions show a lowered threshold to detect anger in violent offenders compared to the general population. This effect is especially pronounced for male faces, correlates with self-reported aggression and presents in absence of a general emotion recognition impairment.ConclusionsThe results indicate that a hostile attribution, related to individual level of aggression and pronounced for male faces, might be one mechanism mediating physical violence.

Study on an effective cross-stimulus emotion recognition model using EEGs based on feature selection and support vector machine

Electroencephalographic (EEG)-based emotion recognition has received increasing attention in the field of human–computer interaction (HCI) recently, there however remain a number of challenges in building a generalized emotion recognition model, one of which includes the difficulty of an EEG-based emotion classifier trained on a specific stimulus to handle other stimuli. Little attention has been paid to this issue. The current paper is to study this issue and determine the feasibility of coping with this challenge using feature selection. 12 healthy volunteers were emotionally elicited when watching the video clip. Power spectral density (PSD) and brain asymmetry (BAY) were extracted as EEG features. Support vector machine (SVM) classifier was then examined under within-stimulus conditions (samples extracted from one video were sent to both training set and testing set) and cross-stimulus conditions (samples extracted from one video were merely sent to one set, training set or testing set alternatively). The within-stimulus 5-class classification performed fairly well (accuracy: 93.31 % for PSD and 85.39 % for BAY). Cross-stimulus classification, however, deteriorated to low levels (46.22 % and 46.2 % accordingly). Trained and tested with the most robust feature subset selected by SVM-recursive feature elimination (RFE), the mean 5-class performance of cross-stimulus classifier was significantly improved to 68.89 and 64.44 % for PSD and BAY respectively. These results suggest that cross-stimulus emotion recognition is reasonable and feasible with proper methods and brings EEG-based emotion recognition models closer to being able to discriminate emotion states in practical application.

Prosody recognition in adults with high-functioning autism spectrum disorders: from psychoacoustics to cognition.

Prosody is an important tool of human communication, carrying both affective and pragmatic messages in speech. Prosody recognition relies on processing of acoustic cues, such as the fundamental frequency of the voice signal, and their interpretation according to acquired socioemotional scripts. Individuals with autism spectrum disorders (ASD) show deficiencies in affective prosody recognition. These deficiencies have been mostly associated with general difficulties in emotion recognition. The current study explored an additional association between affective prosody recognition in ASD and auditory perceptual abilities. Twenty high-functioning male adults with ASD and 32 typically developing male adults, matched on age and verbal abilities undertook a battery of auditory tasks. These included affective and pragmatic prosody recognition tasks, two psychoacoustic tasks (pitch direction recognition and pitch discrimination), and a facial emotion recognition task, representing nonvocal emotion recognition. Compared with controls, the ASD group demonstrated poorer performance on both vocal and facial emotion recognition, but not on pragmatic prosody recognition or on any of the psychoacoustic tasks. Both groups showed strong associations between psychoacoustic abilities and prosody recognition, both affective and pragmatic, although these were more pronounced in the ASD group. Facial emotion recognition predicted vocal emotion recognition in the ASD group only. These findings suggest that auditory perceptual abilities, alongside general emotion recognition abilities, play a significant role in affective prosody recognition in ASD.

Do facial expression recognition deficits in disordered eating reflect a general impairment in attention and executive function?

Impaired facial expression recognition (FER) has been found to be related to clinical and subclinical disordered eating. Impairments in attention and executive function have also been associated with FER deficits in a number of psychological disorders, and with eating psychopathology. The current study aimed to determine whether FER deficits observed in disordered eating reflect more global impairments in cognitive function. Participants with high and low scores on the eating disorders inventory-II (EDI-II) were compared on their ability to recognise emotional facial expressions using the BU-3DFE photographic database. They also completed a battery of neuropsychological tasks on the Cambridge Neuropsychological Test Automated Battery (CANTAB). Relative to the low-EDI group, high-EDI participants accurately recognised significantly more angry and surprised faces. Visual attention was impaired in high-EDI participants. Cognitive flexibility and body dissatisfaction were significant predictors of surprise recognition. Cognitive flexibility explained a significant proportion of variance in general emotion recognition, whilst bulimia significantly predicted anger recognition accuracy. These findings remained significant after controlling for the effects of depression, anxiety and alexithymia. Subsequent analyses revealed that gender differences may explain the differences between current findings and previous evidence.

“Tell me how do I feel” – Emotion recognition and theory of mind in symptomatic mesial temporal lobe epilepsy

Specific interictal personality characteristics in epilepsy, sometimes referred to as “Waxman–Geschwind Syndrome”, have been recognized for centuries and extensively described. Despite the persevering clinical impression that patients with mesial temporal lobe epilepsies (MTLE) suffer from problems in communication and interpersonal relations, uncertainties and controversies remain as to the precise origin of these psychosocial difficulties. Here, we investigated social-cognitive and decision-making abilities using a set of tasks that combine behavioural and psychological measures of social and emotional variables to answer the question of whether patients with MTLE are specifically impaired in social cognition compared to both an epilepsy and a healthy control group. MTLE patients, an epilepsy control group (extra-MTLE; patients with epilepsy, not originating within the frontal or mesial temporal lobe) and healthy controls (HC) were assessed according to their general cognitive status as well as with our Social Cognition Battery, which included tests of basic processes of social cognition, theory of mind, decision making, and various aspects of psychopathology and quality of life. MTLE patients were significantly impaired compared to HC on most measures of the Social Cognition Battery. MTLE patients were predominantly impaired in general emotion recognition compared to extra-MTLE patients. Performance in the epilepsy control group, although not significantly differing from performance in either the MTLE or the healthy control group, lay between these two groups. MTLE can be considered a significant risk factor for the development of deficits in social cognition beyond weaknesses that might be associated with epilepsy as a stigmatized chronic neurological disorder. The presence of deficits in social cognition may explain various behavioural symptoms that have historically driven concepts such as “epileptic personality” or “interictal personality disorder” and may indicate new routes for therapeutic interventions.

The influence of emotion clarity on emotional prosody identification in paranoid schizophrenia

Identification of emotional facial expression and emotional prosody (i.e. speech melody) is often impaired in schizophrenia. For facial emotion identification, a recent study suggested that the relative deficit in schizophrenia is enhanced when the presented emotion is easier to recognize. It is unclear whether this effect is specific to face processing or part of a more general emotion recognition deficit. We used clarity-graded emotional prosodic stimuli without semantic content, and tested 25 in-patients with paranoid schizophrenia, 25 healthy control participants and 25 depressive in-patients on emotional prosody identification. Facial expression identification was used as a control task. Patients with paranoid schizophrenia performed worse than both control groups in identifying emotional prosody, with no specific deficit in any individual emotion category. This deficit was present in high-clarity but not in low-clarity stimuli. Performance in facial control tasks was also impaired, with identification of emotional facial expression being a better predictor of emotional prosody identification than illness-related factors. Of those, negative symptoms emerged as the best predictor for emotional prosody identification. This study suggests a general deficit in identifying high-clarity emotional cues. This finding is in line with the hypothesis that schizophrenia is characterized by high noise in internal representations and by increased fluctuations in cerebral networks.