Emotion Recognition Studies Research Articles

Older adults' emotion recognition: No auditory-visual benefit for less clear expressions.

The ability to recognise emotion from faces or voices appears to decline with advancing age. However, some studies have shown that emotion recognition of auditory-visual (AV) expressions is largely unaffected by age, i.e., older adults get a larger benefit from AV presentation than younger adults resulting in similar AV recognition levels. An issue with these studies is that they used well-recognised emotional expressions that are unlikely to generalise to real-life settings. To examine if an AV emotion recognition benefit generalizes across well and less well recognised stimuli, we conducted an emotion recognition study using expressions that had clear or unclear emotion information for both modalities, or clear visual, but unclear auditory information. Older (n = 30) and younger (n = 30) participants were tested on stimuli of anger, happiness, sadness, surprise, and disgust (expressed in spoken sentences) in auditory-only (AO), visual-only (VO), or AV format. Participants were required to respond by choosing one of 5 emotion options. Younger adults were more accurate in recognising emotions than older adults except for clear VO expressions. Younger adults showed an AV benefit even when unimodal recognition was poor. No such AV benefit was found for older adults; indeed, AV was worse than VO recognition when AO recognition was poor. Analyses of confusion responses indicated that older adults generated more confusion responses that were common between AO and VO conditions, than younger adults. We propose that older adults' poorer AV performance may be due to a combination of weak auditory emotion recognition and response uncertainty that resulted in a higher cognitive load.

Affection of facial artifacts caused by micro-expressions on electroencephalography signals.

Macro-expressions are widely used in emotion recognition based on electroencephalography (EEG) because of their use as an intuitive external expression. Similarly, micro-expressions, as suppressed and brief emotional expressions, can also reflect a person's genuine emotional state. Therefore, researchers have started to focus on emotion recognition studies based on micro-expressions and EEG. However, compared to the effect of artifacts generated by macro-expressions on the EEG signal, it is not clear how artifacts generated by micro-expressions affect EEG signals. In this study, we investigated the effects of facial muscle activity caused by micro-expressions in positive emotions on EEG signals. We recorded the participants' facial expression images and EEG signals while they watched positive emotion-inducing videos. We then divided the 13 facial regions and extracted the main directional mean optical flow features as facial micro-expression image features, and the power spectral densities of theta, alpha, beta, and gamma frequency bands as EEG features. Multiple linear regression and Granger causality test analyses were used to determine the extent of the effect of facial muscle activity artifacts on EEG signals. The results showed that the average percentage of EEG signals affected by muscle artifacts caused by micro-expressions was 11.5%, with the frontal and temporal regions being significantly affected. After removing the artifacts from the EEG signal, the average percentage of the affected EEG signal dropped to 3.7%. To the best of our knowledge, this is the first study to investigate the affection of facial artifacts caused by micro-expressions on EEG signals.

A Pattern Mining Approach for Improving Speech Emotion Recognition

Speech-driven user interfaces are becoming more common in our lives. To interact with such systems naturally and effectively, machines need to recognize the emotional states of users and respond to them accordingly. At the heart of the emotion recognition research done to this end lies the emotion representation that enables machines to learn and predict emotions. Speech emotion recognition studies use a wide range of low-to-high-level acoustic features for representation purposes such as LLDs, their functionals, and BoAW. In this paper, we present a new method for extracting a novel set of high-level features for classifying emotions. For this purpose, we (1) reduce the dimension of discrete-time speech signals, (2) perform a quantization operation on the new signals and assign a distinct symbol to each quantization level, (3) use the symbol sequences representing the signals to extract discriminative patterns that are capable of distinguishing different emotions from each other, and (4) generate a separate set of features for each emotion from the extracted patterns. Experimental results show that pattern features outperform Energy, Voicing, MFCC, Spectral, and RASTA feature sets. We also demonstrate that combining the pattern-based features and the acoustic features further improves the classification performance.

Subject independent emotion recognition using EEG and physiological signals – a comparative study

PurposeThe aim of this study is to investigate subject independent emotion recognition capabilities of EEG and peripheral physiological signals namely: electroocoulogram (EOG), electromyography (EMG), electrodermal activity (EDA), temperature, plethysmograph and respiration. The experiments are conducted on both modalities independently and in combination. This study arranges the physiological signals in order based on the prediction accuracy obtained on test data using time and frequency domain features.Design/methodology/approachDEAP dataset is used in this experiment. Time and frequency domain features of EEG and physiological signals are extracted, followed by correlation-based feature selection. Classifiers namely – Naïve Bayes, logistic regression, linear discriminant analysis, quadratic discriminant analysis, logit boost and stacking are trained on the selected features. Based on the performance of the classifiers on the test set, the best modality for each dimension of emotion is identified.Findings The experimental results with EEG as one modality and all physiological signals as another modality indicate that EEG signals are better at arousal prediction compared to physiological signals by 7.18%, while physiological signals are better at valence prediction compared to EEG signals by 3.51%. The valence prediction accuracy of EOG is superior to zygomaticus electromyography (zEMG) and EDA by 1.75% at the cost of higher number of electrodes. This paper concludes that valence can be measured from the eyes (EOG) while arousal can be measured from the changes in blood volume (plethysmograph). The sorted order of physiological signals based on arousal prediction accuracy is plethysmograph, EOG (hEOG + vEOG), vEOG, hEOG, zEMG, tEMG, temperature, EMG (tEMG + zEMG), respiration, EDA, while based on valence prediction accuracy the sorted order is EOG (hEOG + vEOG), EDA, zEMG, hEOG, respiration, tEMG, vEOG, EMG (tEMG + zEMG), temperature and plethysmograph.Originality/valueMany of the emotion recognition studies in literature are subject dependent and the limited subject independent emotion recognition studies in the literature report an average of leave one subject out (LOSO) validation result as accuracy. The work reported in this paper sets the baseline for subject independent emotion recognition using DEAP dataset by clearly specifying the subjects used in training and test set. In addition, this work specifies the cut-off score used to classify the scale as low or high in arousal and valence dimensions. Generally, statistical features are used for emotion recognition using physiological signals as a modality, whereas in this work, time and frequency domain features of physiological signals and EEG are used. This paper concludes that valence can be identified from EOG while arousal can be predicted from plethysmograph.

Electroencephalography based emotion detection using ensemble classification and asymmetric brain activity

Over the past decade, emotion detection using rhythmic brain activity has become a critical area of research. The asymmetrical brain activity has garnered the most significant level of research attention due to its implications for the study of emotions, including hemispheric asymmetry or, more generally, asymmetrical brain activity. This study aimed at enhancing the accuracy of emotion detection using Electroencephalography (EEG) brain signals. This happens by identifying electrodes where relevant brain activity changes occur during the emotions and by defining pairs of relevant electrodes having asymmetric brain activities during emotions. Experimental results showed that the proposed method is highly competitive compared with existing studies of multi-class emotion recognition. These results were improved by processing not the whole EEG signals but by focusing on fragments of the signals, called epochs, which represent the instants where the excitation is maximum during emotions. The epochs were extracted using the zero-time windowing method and the numerator group-delay function.

Multi-source joint domain adaptation for cross-subject and cross-session emotion recognition from electroencephalography.

As an important component to promote the development of affective brain–computer interfaces, the study of emotion recognition based on electroencephalography (EEG) has encountered a difficult challenge; the distribution of EEG data changes among different subjects and at different time periods. Domain adaptation methods can effectively alleviate the generalization problem of EEG emotion recognition models. However, most of them treat multiple source domains, with significantly different distributions, as one single source domain, and only adapt the cross-domain marginal distribution while ignoring the joint distribution difference between the domains. To gain the advantages of multiple source distributions, and better match the distributions of the source and target domains, this paper proposes a novel multi-source joint domain adaptation (MSJDA) network. We first map all domains to a shared feature space and then align the joint distributions of the further extracted private representations and the corresponding classification predictions for each pair of source and target domains. Extensive cross-subject and cross-session experiments on the benchmark dataset, SEED, demonstrate the effectiveness of the proposed model, where more significant classification results are obtained on the more difficult cross-subject emotion recognition task.

Kind of Blue

In this article, Michelle Henning considers the use of emotion recognition and sentiment analysis on social media photographs as part of a regime of surveillance and control. Automated emotion recognition studies reduce social media photographs to data, reading them as symptoms of mental states. This goes against the historical idea of the photograph as objective and detached, but corresponds with a contemporary therapeutic sense of self. Henning acknowledges that people use social photos expressively, but argues that emotion recognition techniques rely on a behaviourist and positivist model that is indifferent to intentions and fails to take culture into account.

Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis

Multimodal Emotion Recognition for Conversation (ERC) is a challenging multi-class classification task that requires recognizing multiple speakers’ emotions in text, audio, video, and other modalities. ERC has received considerable attention from researchers due to its potential applications in opinion mining, advertising, and healthcare. However, the syntactic structure characteristics of the text itself have not been considered in this study. Taking into account this, this paper proposes a conversational affective analysis model (DSAGCN) combining dependent syntactic analysis and graph convolutional neural networks. Since words that reflect emotional polarity are usually concentrated exclusively in limited regions, the DSAGCN model first employs a self-attention mechanism to capture the most effective words in the dialogue context and obtain a more accurate vector representation of the emotional semantics. Then, based on speaker relationships and dependent syntactic relationships, the multimodal sentiment relationship graphs are constructed. Finally, a graph convolutional neural network is used to complete the recognition of multimodal emotion. In extensive experiments on two real datasets, IEMOCAP and MELD, the DSAGCN model outperforms the existing models in terms of average accuracy and f1 values for multimodal emotion recognition, especially for emotions such as “happiness” and “anger”. Thus, dependent syntactic analysis and self-attention mechanism can enhance the model’s ability to understand emotions.

Digital emotion analytics: a pilot study of human emotion recognition using mobile device sensors

Affective computing is a rapidly developing area at the intersection of psychology and the development of artificial intelligence systems. At the moment, systems for recognising human emotions from photos and videos (facial expressions), voice recordings (intonation), gestures, posture, gait and other data using various machine learning algorithms are actively developed. This research project is devoted to the recognition of emotions and mental states of a person through the sensors of mobile devices (accelerometer, gyroscope, etc.), which record the features of micro- and macro-hand motor skills characteristic of the states under study. The pilot study considered the possibility of fixing and differentiating psychoemotional states according to the readings of sensors of mobile devices (tablet, smartphone), using machine learning models. As a result, it was possible to obtain models that determine, according to the readings of the sensors of a mobile device in the hands of a person, whether it is in a neutral emotional state or under stress. Moreover, it was possible to differentiate the state of stress according to two modalities – stress caused by psychological reasons ("fulfillment of obligations") and psychophysiological reasons (unpleasant noise in the headphones). The statistically significant differences, as well as the relatively high accuracy of the constructed machine learning model, allow us to speak about the reliability of the results obtained, and they confirm the hypothesis about the possibility of identifying and classifying emotional states using the sensors of mobile devices.

ASERS-CNN: Arabic Speech Emotion Recognition System based on CNN Model

When two people are on the phone, although they cannot observe the other person's facial expression and physiological state, it is possible to estimate the speaker's emotional state by voice roughly. In medical care, if the emotional state of a patient, especially a patient with an expression disorder, can be known, different care measures can be made according to the patient's mood to increase the amount of care. The system that capable for recognize the emotional states of human being from his speech is known as Speech emotion recognition system (SER). Deep learning is one of most technique that has been widely used in emotion recognition studies, in this paper we implement CNN model for Arabic speech emotion recognition. We propose ASERS-CNN model for Arabic Speech Emotion Recognition based on CNN model. We evaluated our model using Arabic speech dataset named Basic Arabic Expressive Speech corpus (BAES-DB). In addition of that we compare the accuracy between our previous ASERS-LSTM and new ASERS-CNN model proposed in this paper and we comes out that our new proposed mode is outperformed ASERS-LSTM model where it get 98.18% accuracy.

A Dataset for Emotion Recognition Using Virtual Reality and EEG (DER-VREEG): Emotional State Classification Using Low-Cost Wearable VR-EEG Headsets

Emotions are viewed as an important aspect of human interactions and conversations, and allow effective and logical decision making. Emotion recognition uses low-cost wearable electroencephalography (EEG) headsets to collect brainwave signals and interpret these signals to provide information on the mental state of a person, with the implementation of a virtual reality environment in different applications; the gap between human and computer interaction, as well as the understanding process, would shorten, providing an immediate response to an individual’s mental health. This study aims to use a virtual reality (VR) headset to induce four classes of emotions (happy, scared, calm, and bored), to collect brainwave samples using a low-cost wearable EEG headset, and to run popular classifiers to compare the most feasible ones that can be used for this particular setup. Firstly, we attempt to build an immersive VR database that is accessible to the public and that can potentially assist with emotion recognition studies using virtual reality stimuli. Secondly, we use a low-cost wearable EEG headset that is both compact and small, and can be attached to the scalp without any hindrance, allowing freedom of movement for participants to view their surroundings inside the immersive VR stimulus. Finally, we evaluate the emotion recognition system by using popular machine learning algorithms and compare them for both intra-subject and inter-subject classification. The results obtained here show that the prediction model for the four-class emotion classification performed well, including the more challenging inter-subject classification, with the support vector machine (SVM Class Weight kernel) obtaining 85.01% classification accuracy. This shows that using less electrode channels but with proper parameter tuning and selection features affects the performance of the classifications.

Bangla Speech Emotion Recognition and Cross-Lingual Study Using Deep CNN and BLSTM Networks

In this study, we have presented a deep learning-based implementation for speech emotion recognition (SER). The system combines a deep convolutional neural network (DCNN) and a bidirectional long-short term memory (BLSTM) network with a time-distributed flatten (TDF) layer. The proposed model has been applied for the recently built audio-only Bangla emotional speech corpus SUBESCO. A series of experiments were carried out to analyze all the models discussed in this paper for baseline, cross-lingual, and multilingual training-testing setups. The experimental results reveal that the model with a TDF layer achieves better performance compared with other state-of-the-art CNN-based SER models which can work on both temporal and sequential representation of emotions. For the cross-lingual experiments, cross-corpus training, multi-corpus training, and transfer learning were employed for the Bangla and English languages using the SUBESCO and RAVDESS datasets. The proposed model has attained a state-of-the-art perceptual efficiency achieving weighted accuracies (WAs) of 86.9%, and 82.7% for the SUBESCO and RAVDESS datasets, respectively.

EEG-Based Emotion Recognition With Haptic Vibration by a Feature Fusion Method

Emotion recognition based on electroencephalogram (EEG) signals has been one of the most active research topics of affective computing. In previous studies of emotion recognition, the selection of stimulus sources was usually focused on single stimuli, such as visual or auditory. In this work, we propose a novel emotional stimulation scheme that synchronizes haptic vibration with audiovisual content to form a mixed sense of visual–auditory–haptic to trigger emotions. Fifteen subjects were recruited to watch the four kinds of emotional movie clips (happiness, fear, sadness, and neutral) with haptic or not, and their EEG signals were collected simultaneously. The power spectral density (PSD) feature, differential entropy (DE) feature, wavelet entropy (WE) feature, and brain function network (BFN) feature were extracted and fused to reflect the time–frequency–spatial domain of emotional EEG signals. The t-distributed stochastic neighbor embedding (t-SNE) was utilized for dimensionality reduction and feature selection. In addition, the fusion features are classified by the stacking ensemble learning framework. The experimental results show that the proposed haptic vibration strategy can enhance the activity of emotion-related brain regions, and the average classification accuracy was 85.46%.

Comparative study of emotion recognition and theory of mind between major depressive disorder and schizophrenia

BackgroundSocial cognition as a transdiagnostic construct between major depressive disorder (MDD) and schizophrenia (SCZ) is not well understood. This may be attributed to the variability of social cognitive measures indexing the same construct. This study aims to compare emotion recognition and theory of mind domains, known to be impaired in SCZ, between MDD and SCZ. MethodsThree groups of participants (NTotal = 150) were enrolled in this study: MDD (n = 51), SCZ (n = 50) and healthy controls (HC; n = 49). Emotion recognition was assessed on the Bell Lysaker Emotion Recognition Task (BLERT) and Penn Emotion Recognition Task (ER40); theory of mind was measured on The Awareness of Social Inference Test (TASIT). Mixed ANCOVAs were utilised to compare social cognitive performance across the groups. ResultsSCZ performed poorer in all 3 social cognition tasks compared to both MDD and HC. No statistically significant difference in social cognitive performance was observed between MDD and HC. ConclusionsThis study serves as an effort towards employing the same standardised social cognitive measures for direct comparison of performance patterns across diagnostic groups. Future work is needed to extend this in larger samples of different illness severity and diagnostic categories.

Brain lateralisation feature extraction and ant colony optimisation‐bidirectional LSTM network model for emotion recognition

The human inner perception is the core technology for human-robot interactions. Emotion recognition has been established as a representative study for recognising the internal state of human beings in addition to research regarding intention recognition. In the study of emotion recognition using EEG signals, emotion evaluation is categorised into two methods: discrete emotion and continuous emotion. This study proposes an ant colony optimisation-bidirectional LSTM network model. Unlike other LSTM network models, this model improves performance by applying more weight values that are valid for emotion recognition in the current LSTM cell state using past and future biosignal information and combining ACO to find the optimal combination of emotion recognition features among many features. Furthermore, it extracts valid features using peripheral nervous system signals PPG, GSR, and EOG as well as the central nervous system signals EEG, simultaneously. The authors reinforce feature performance by adding brain lateralisation for emotion recognition. Emotion label data were recorded by performing annotation labelling, in which the values are between −100 and 100 in the arousal-valence domain. The experimental result shows that the ACO-bidirectional LSTM model using brain lateralisation in the MAHNOB-HCI, DEAP, MERTI-Apps database yields the best valence performance, RMSE of 0.0442, 0.0523, and 0.0568, respectively.

CNN based efficient approach for emotion recognition

Determining the psychophysiological state of people has been a significant issue in many fields, such as the adaptation of disabled people to social life. Recently, various physiological signals have been used in emotion recognition studies since the desired success is limited in studies conducted with traditional methods such as posture and facial expressions. In this study, a GoogLeNet-based deep learning method that can automatically detect human emotions using Electroencephalogram (EEG) signals is proposed. EEG signals were converted to EEG images (scalogram) with Continuous Wavelet Transform (CWT), which is more sensitive to Time Frequency (TF) changes in EEG signals. Then, feature extraction was performed from EEG images with a pre-trained GoogLeNet. Finally, the deep features obtained were applied to popular machine learning methods such as k-Nearest Neighbor (k-NN), Support Vector Machine (SVM), and Extreme Learning Machine (ELM) classifiers for emotion classification. The proposed method was tested using the GAMEEMO dataset. This study regards two emotions, namely ‘Positive’ and ‘Negative’ for classification. The comprehensive experimental results show that the proposed method achieved 98.78%, 98.53%, and 98.41% emotion detection accuracy in SVM, k-NN, and ELM classifiers, respectively. The proposed method shows improved performance of about 21.85% than of the existing state-of-the-art methods using the same dataset.

Research on Emotion Recognition and Dementias: Foundations and Prospects.

The study of emotion recognition could be crucial for detecting alterations in certain cognitive areas or as an early sign of neurological disorders. The main objective of the study is to characterize research development on emotion recognition, identifying the intellectual structure that supports this area of knowledge, and the main lines of research attracting investigators' interest. We identified publications on emotion recognition and dementia included in the Web of Science Core Collection, analyzing the scientific output and main disciplines involved in generating knowledge in the area. A co-citation analysis and an analysis of the bibliographic coupling between the retrieved documents elucidated the thematic orientations of the research and the reference works that constitute the foundation for development in the field. A total of 345 documents, with 24,282 bibliographic references between them, were included. This is an emerging research area, attracting the interest of investigators in Neurosciences, Psychology, Clinical Neurology, and Psychiatry, among other disciplines. Four prominent topic areas were identified, linked to frontotemporal dementia, autism spectrum disorders, Alzheimer's disease, and Parkinson's and Huntington disease. Many recent papers focus on the detection of mild cognitive impairment. Impaired emotion recognition may be a key sign facilitating the diagnosis and early treatment of different neurodegenerative diseases as well as for triggering the necessary provision of social and family support, explaining the growing research interest in this area.

Brain network features based on theta-gamma cross-frequency coupling connections in EEG for emotion recognition

Emotion recognition is a hot topic in the field of cognitive neuroscience and interpersonal interaction, and EEG feature selection is an important classification technology. At present, the mainstream method of EEG feature selection is to extract non-interactive features of channels such as power spectral density, or correlation features among local multi-channels. With the application of complex network graph theory, the connection network between multiple brain regions is gradually included in feature selection. However, in the process of brain network construction, most of the current connections adopt simple signal phase or amplitude synchronization. In recent years, it has been found that in the process of emotion, memory, learning, and other advanced cognitive processes, the large-scale connection and communication between the brain regions are mainly completed by the cross-frequency coupling(CFC) between the low-frequency phase and the high-frequency amplitude of neural oscillations. Based on this, we use CFC to update the connection mode, reconstruct the brain network, and extract features for emotion recognition research. Our results show that the EEG network based on CFC performs better than other EEG synchronization networks in emotion classification. Moreover, the combination of global features and local features of the brain network, as well as the dynamic network features with continuous time-windows, can effectively improve the accuracy of emotion recognition. This study provides a new idea of network connection for the follow-up study of emotion recognition and other advanced cognitive activities and makes a pioneering exploration for further research on feature selection of emotion recognition and related neural circuits at the brain network level of functional connectivity.

Convolutional Neural Networks Model for Emotion Recognition Using EEG Signal

A Brain-computer interface (BCI) using an electroencephalogram (EEG) signal has a great attraction in emotion recognition studies due to its resistance to humans’ deceptive actions. This is the most significant advantage of brain signals over speech or visual signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that a lot of effort is required for manually feature extractor, EEG recordings show varying distributions for different people and the same person at different time instances. The Poor generalization ability of the network model as well as low robustness of the recognition system. Improving algorithms and machine learning technology helps researchers to recognize emotion easily. In recent years, deep learning (DL) techniques, specifically convolutional neural networks (CNNs) have made excellent progress in many applications. This study aims to reduce the manual effort on features extraction and improve the EEG signal single model’s emotion recognition using convolutional neural network (CNN) architecture with residue block. The dataset is shuffle, divided into training and testing, and then fed to the model. DEAP dataset has class 1, class 2, class 3, and class 4 for both valence and arousal with an accuracy of 90.69%, 91.21%, 89.66%, 93.64% respectively, with a mean accuracy of 91.3%. The negative emotion has the highest accuracy of 94.86% fellow by neutral emotion with 94.29% and positive emotion with 93.25% respectively, with a mean accuracy of 94.13% on the SEED dataset. The experimental results indicated that CNN Based on residual networks can achieve an excellent result with high recognition accuracy, which is superior to most recent approaches.

Emotion detection using electroencephalography signals and a zero-time windowing-based epoch estimation and relevant electrode identification

Recognizing emotions using biological brain signals requires accurate and efficient signal processing and feature extraction methods. Existing methods use several techniques to extract useful features from a fixed number of electroencephalography (EEG) channels. The primary objective of this study was to improve the performance of emotion recognition using brain signals by applying a novel and adaptive channel selection method that acknowledges that brain activity has a unique behavior that differs from one person to another and one emotional state to another. Moreover, we propose identifying epochs, which are the instants at which excitation is maximum, during the emotion to improve the system’s accuracy. We used the zero-time windowing method to extract instantaneous spectral information using the numerator group-delay function to accurately detect the epochs in each emotional state. Different classification scheme were defined using QDC and RNN and evaluated using the DEAP database. The experimental results showed that the proposed method is highly competitive compared with existing studies of multi-class emotion recognition. The average accuracy rate exceeded 89%. Compared with existing algorithms dealing with 9 emotions, the proposed method enhanced the accuracy rate by 8%. Moreover, experiment shows that the proposed system outperforms similar approaches discriminating between 3 and 4 emotions only. We also found that the proposed method works well, even when applying conventional classification algorithms.