Emotion Recognition Experiments Research Articles

Featural vs. Holistic processing and visual sampling in the influence of social category cues on emotion recognition

ABSTRACT Past research demonstrates that emotion recognition is influenced by social category cues present on faces. However, little research has investigated whether holistic processing is required to observe these influences of social category information on emotion perception, and no studies have investigated whether different visual sampling strategies (i.e. differences in the allocation of attention to different regions of the face) contribute to the interaction between social cues and emotional expressions. The current study aimed to address this. Participants categorised happy and angry expressions on own- and other-race faces, and male and female faces. In Experiments 1 and 2, holistic processing was disrupted by presenting inverted faces (Experiment 1) or part faces (Experiment 2). In Experiments 3 and 4 participants’ eye-gaze to eye and mouth regions was also tracked. Disrupting holistic processing did not alter the moderating influence of sex and race cues on emotion recognition (Experiments 1, 2, 4). Gaze patterns differed as a function of emotional expression, and social category cues, however, eye-gaze patterns did not reflect response time patterns (Experiments 3 and 4). Results indicate that the interaction between social category cues and emotion does not require holistic processing and is not driven by differences in visual sampling.

MULTI-FEATURE FUSION EMOTION RECOGNITION BASED ON RESTING EEG

An important task of brain–computer interface (BCI) research is to read or decode mental content from a large number of electroencephalographic (EEG) signals, which is an exciting, attractive, and challenging goal. In this paper, resting EEG data are collected from spontaneous EEG, and 16 random features such as correlation coefficient, covariance, and brainpower spectrum are extracted as reference vectors. In the subsequent emotion recognition experiment, the same feature information is extracted and separated from the EEG signal, and the translation and normalization processing are carried out based on the resting-state features. Finally, with the machine learning methods such as [Formula: see text]-means clustering and multi-feature fusion, the positive, negative, and neutral emotional characteristic parameters were correctly separated. In a group of 12 subjects, the correct recognition rate of visual evoked positive, negative, and neutral emotions reached 83.9%, which was better than the literature mentioned in this paper. Another highlight of this method is that it can quickly, accurately, and efficiently select the number of features with the best matching and the least resource consumption from multiple features and multiple potential acquisition points. Further analysis and comparison of EEG characteristics can find the relationship between specific stimuli and corresponding EEG characteristic signals.

Improvement of multilingual emotion recognition method based on normalized acoustic features using CRNN

Several speech emotion recognition systems have been proposed, mostly targeting a single language; however, the recognition performance is significantly degraded for languages other than the target language. To solve this problem, a method of normalizing features using synthesized speech was proposed by Okabe et al.; however, this method requires a search for the optimal normalization operation.In this paper, we propose a multilingual emotion recognition method that automatically normalizes acoustic features and recognizes emotions using the “recognition rate maximization” criterion. Synthesized speech with the same content as emotional speech was generated and input to a convolutional recurrent neural network with time axis aligned. Feature normalization by comparison with synthesized speech and emotion recognition were performed in a single network, and efficient training was expected.In speech emotion recognition experiments with four languages, normalization improved the recognition accuracy from 47.5% to 49.0%. Because normalization was not effective for utterances in which the speaking rate of the synthesized and emotional speeches differed significantly, we proposed a method to adjust the speaking rate of the synthesized speech and further improved the accuracy by up to 50.4%.

Deep Neural Network for Emotion Recognition Based on Meta-Transfer Learning

In recent years, many EEG-based emotion recognition methods have been proposed, which can achieve good performance on single-subject data. However, when the models are applied to cross-subject scenarios, due to the existence of subject differences, these models are often difficult to accurately identify the emotions of new subjects, which is not conducive to the practical application of the models. Many transfer learning methods have been applied to cross-subject EEG emotion recognition tasks to reduce the effect of subject differences. Most of them need to be trained with source data of many subjects and calibrated with more data of target subjects to obtain better classification performance on target subjects. However, this process relies on a large amount of training data to guarantee the final effect. This paper proposed a meta-transfer learning model for emotion recognition. The model can reduce the amount of data required by the meta-learning optimization algorithm. Even if only a small amount of data is used for training, it can achieve good performance, thereby reducing the cost of EEG acquisition and labeling, and it is also conducive to the model for new subjects. Finally, this paper conducts cross-subject emotion recognition experiments based on two public datasets SEED and SEED-IV. The experimental results show that the performance of the proposed meta-transfer learning method is better than the baseline method, and can rapid adaptation to unknown subjects while reducing training data.

Personality first in emotion: a deep neural network based on electroencephalogram channel attention for cross-subject emotion recognition.

In recent years, more and more researchers have focused on emotion recognition methods based on electroencephalogram (EEG) signals. However, most studies only consider the spatio-temporal characteristics of EEG and the modelling based on this feature, without considering personality factors, let alone studying the potential correlation between different subjects. Considering the particularity of emotions, different individuals may have different subjective responses to the same physical stimulus. Therefore, emotion recognition methods based on EEG signals should tend to be personalized. This paper models the personalized EEG emotion recognition from the macro and micro levels. At the macro level, we use personality characteristics to classify the individuals’ personalities from the perspective of ‘birds of a feather flock together’. At the micro level, we employ deep learning models to extract the spatio-temporal feature information of EEG. To evaluate the effectiveness of our method, we conduct an EEG emotion recognition experiment on the ASCERTAIN dataset. Our experimental results demonstrate that the recognition accuracy of our proposed method is 72.4% and 75.9% on valence and arousal, respectively, which is 10.2% and 9.1% higher than that of no consideration of personalization.

Feature and channel selection for designing a regression-based continuous-variable emotion recognition system with two EEG channels

Objective:With deepened interactions between human and computer, the need for a reliable and practical system for emotion recognition has become significant. The aim of this study is to propose a practical system for estimation of a continuous measure of valence based on a few number of EEG channels. Methods:A vast spectrum of time, frequency and coherence features were implemented with linear Regression (LR), Support Vector Regression (SVR) and Multi-Layer Perceptron (MLP) models and then ranked for the performance on DEAP database using a regression-based Relief filter. Regression outcomes were also classified to compare the performance of the proposed method with the literature. Finally, a video-based emotion recognition experiment was designed and conducted on 12 subjects using F7, F8, FC2 and T7 electrodes. Results:Magnitude Squared Coherence Estimate(MSCE) on F7–F8 with SVR model provided the highest performance on DEAP dataset. Classification of the output led to an average accuracy of 67.5%. For the gathered data, combination of MSCE and Hilbert–Huang Spectrum provided the best performance with 0.22 root mean square error and 0.67 correlation with self-reported valence in the scale of 1–9. Conclusion:MSCE could provide a good accuracy in estimation of Valence using 2 EEG channels on Deep dataset, and with addition of Hilbert–Huang Spectrum, it also demonstrated good accuracy and correlation with self-reported valence, in a completely different experiment. Significance:Continuous-value estimation of the valence can be achieved with only 2 EEG channels for practical applications out of the lab.

Analysis of Emotion Recognition Model Using Electroencephalogram (EEG) Signals Based on Stimuli Text

Recognizing emotions through the brain wave approach with facial or sound expression is widely used, but few use text stimuli. Therefore, this study aims to analyze the emotion recognition experiment by stimulating sentiment-tones using EEG. The process of classifying emotions uses a random forest model approach which is compared with two models, namely Support Vector Machine and decision tree as benchmarks. The raw data used comes from the results of scrapping Twitter data. The dataset of emotional annotation was carried out manually based on four classifications, specifically: happiness, sadness, fear, and anger. The annotated dataset was tested using an Electroencephalogram (EEG) device attached to the participant's head to determine the brain waves appearing after reading the text. The results showed that the random forest model has the highest accuracy level with a rate of 98% which is slightly different from the decision tree with 88%. Meanwhile, in SVM the accuracy results are less good with a rate of 32%. Furthermore, the match level of angry emotions from the three models above during manual annotation and using the EEG device showed a high number with an average value above 90%, because reading with angry expressions is easier to perform.
 For this reason, this study aims to test the emotion recognition experiment by stimulating sentiment-tones using EEG. The process of classifying emotions uses a random forest model approach which is compared with two models, namely SVM and decision tree as benchmarks. The dataset used comes from the results of scrapping Twitter data.

Electroencephalogram Access for Emotion Recognition Based on a Deep Hybrid Network.

In the human-computer interaction (HCI), electroencephalogram (EEG) access for automatic emotion recognition is an effective way for robot brains to perceive human behavior. In order to improve the accuracy of the emotion recognition, a method of EEG access for emotion recognition based on a deep hybrid network was proposed in this paper. Firstly, the collected EEG was decomposed into four frequency band signals, and the multiscale sample entropy (MSE) features of each frequency band were extracted. Secondly, the constructed 3D MSE feature matrices were fed into a deep hybrid network for autonomous learning. The deep hybrid network was composed of a continuous convolutional neural network (CNN) and hidden Markov models (HMMs). Lastly, HMMs trained with multiple observation sequences were used to replace the artificial neural network classifier in the CNN, and the emotion recognition task was completed by HMM classifiers. The proposed method was applied to the DEAP dataset for emotion recognition experiments, and the average accuracy could achieve 79.77% on arousal, 83.09% on valence, and 81.83% on dominance. Compared with the latest related methods, the accuracy was improved by 0.99% on valence and 14.58% on dominance, which verified the effectiveness of the proposed method.

A Study of Cross-Linguistic Speech Emotion Recognition Based on 2D Feature Spaces

In this research, a study of cross-linguistic speech emotion recognition is performed. For this purpose, emotional data of different languages (English, Lithuanian, German, Spanish, Serbian, and Polish) are collected, resulting in a cross-linguistic speech emotion dataset with the size of more than 10.000 emotional utterances. Despite the bi-modal character of the databases gathered, our focus is on the acoustic representation only. The assumption is that the speech audio signal carries sufficient emotional information to detect and retrieve it. Several two-dimensional acoustic feature spaces, such as cochleagrams, spectrograms, mel-cepstrograms, and fractal dimension-based space, are employed as the representations of speech emotional features. A convolutional neural network (CNN) is used as a classifier. The results show the superiority of cochleagrams over other feature spaces utilized. In the CNN-based speaker-independent cross-linguistic speech emotion recognition (SER) experiment, the accuracy of over 90% is achieved, which is close to the monolingual case of SER.

Emotion Recognition Based on Multiple Physiological Signals

Emotion is a series of reactions triggered by a specific object or situation that affects a person's physiological state and can, therefore, be identified by physiological signals. This paper proposes an emotion recognition model. Extracted the features of physiological signals such as photoplethysmography, galvanic skin response, respiration amplitude, and skin temperature. The SVM-RFE-CBR(Recursive Feature Elimination-Correlation Bias Reduction-Support Vector Machine) algorithm was performed to select features and support vector machines for classification. Finally, the model was implemented on the DEAP dataset for an emotion recognition experiment. In the rating scale of valence, arousal, and dominance, the accuracy rates of 73.5%, 81.3%, and 76.1% were obtained respectively. The result shows that emotional recognition can be effectively performed by combining a variety of physiological signals.

Wearable Emotion Recognition Using Heart Rate Data from a Smart Bracelet.

Emotion recognition and monitoring based on commonly used wearable devices can play an important role in psychological health monitoring and human-computer interaction. However, the existing methods cannot rely on the common smart bracelets or watches for emotion monitoring in daily life. To address this issue, our study proposes a method for emotional recognition using heart rate data from a wearable smart bracelet. A ‘neutral + target’ pair emotion stimulation experimental paradigm was presented, and a dataset of heart rate from 25 subjects was established, where neutral plus target emotion (neutral, happy, and sad) stimulation video pairs from China’s standard Emotional Video Stimuli materials (CEVS) were applied to the recruited subjects. Normalized features from the data of target emotions normalized by the baseline data of neutral mood were adopted. Emotion recognition experiment results approved the effectiveness of ‘neutral + target’ video pair simulation experimental paradigm, the baseline setting using neutral mood data, and the normalized features, as well as the classifiers of Adaboost and GBDT on this dataset. This method will promote the development of wearable consumer electronic devices for monitoring human emotional moods.

From Regional to Global Brain: A Novel Hierarchical Spatial-Temporal Neural Network Model for EEG Emotion Recognition

In this paper, we propose a novel Electroencephalograph (EEG) emotion recognition method inspired by neuroscience with respect to the brain response to different emotions. The proposed method, denoted by R2G-STNN, consists of spatial and temporal neural network models with regional to global hierarchical feature learning process to learn discriminative spatial-temporal EEG features. To learn the spatial features, a bidirectional long short term memory (BiLSTM) network is adopted to capture the intrinsic spatial relationships of EEG electrodes within brain region and between brain regions, respectively. Considering that different brain regions play different roles in the EEG emotion recognition, a region-attention layer into the R2G-STNN model is also introduced to learn a set of weights to strengthen or weaken the contributions of brain regions. Based on the spatial feature sequences, BiLSTM is adopted to learn both regional and global spatial-temporal features and the features are fitted into a classifier layer for learning emotion-discriminative features, in which a domain discriminator working corporately with the classifier is used to decrease the domain shift between training and testing data. Finally, to evaluate the proposed method, we conduct both subject-dependent and subject-independent EEG emotion recognition experiments on SEED database, and the experimental results show that the proposed method achieves state-of-the-art performance.

A comprehensive study on bilingual and multilingual speech emotion recognition using a two-pass classification scheme.

Emotion recognition plays an important role in human-computer interaction. Previously and currently, many studies focused on speech emotion recognition using several classifiers and feature extraction methods. The majority of such studies, however, address the problem of speech emotion recognition considering emotions solely from the perspective of a single language. In contrast, the current study extends monolingual speech emotion recognition to also cover the case of emotions expressed in several languages that are simultaneously recognized by a complete system. To address this issue, a method, which provides an effective and powerful solution to bilingual speech emotion recognition, is proposed and evaluated. The proposed method is based on a two-pass classification scheme consisting of spoken language identification and speech emotion recognition. In the first pass, the language spoken is identified; in the second pass, emotion recognition is conducted using the emotion models of the language identified. Based on deep learning and the i-vector paradigm, bilingual emotion recognition experiments have been conducted using the state-of-the-art English IEMOCAP (four emotions) and German FAU Aibo (five emotions) corpora. Two classifiers along with i-vector features were used and compared, namely, fully connected deep neural networks (DNN) and convolutional neural networks (CNN). In the case of DNN, 64.0% and 61.14% unweighted average recalls (UARs) were obtained using the IEMOCAP and FAU Aibo corpora, respectively. When using CNN, 62.0% and 59.8% UARs were achieved in the case of the IEMOCAP and FAU Aibo corpora, respectively. These results are very promising, and superior to those obtained in similar studies on multilingual or even monolingual speech emotion recognition. Furthermore, an additional baseline approach for bilingual speech emotion recognition was implemented and evaluated. In the baseline approach, six common emotions were considered, and bilingual emotion models were created, trained on data from the two languages. In this case, 51.2% and 51.5% UARs for six emotions were obtained using DNN and CNN, respectively. The results using the baseline method were reasonable and promising, showing the effectiveness of using i-vectors and deep learning in bilingual speech emotion recognition. On the other hand, the proposed two-pass method based on language identification showed significantly superior performance. Furthermore, the current study was extended to also deal with multilingual speech emotion recognition using corpora collected under similar conditions. Specifically, the English IEMOCAP, the German Emo-DB, and a Japanese corpus were used to recognize four emotions based on the proposed two-pass method. The results obtained were very promising, and the differences in UAR were not statistically significant compared to the monolingual classifiers.

SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG.

EEG-based automatic emotion recognition can help brain-inspired robots in improving their interactions with humans. This paper presents a novel framework for emotion recognition using multi-channel electroencephalogram (EEG). The framework consists of a linear EEG mixing model and an emotion timing model. Our proposed framework considerably decomposes the EEG source signals from the collected EEG signals and improves classification accuracy by using the context correlations of the EEG feature sequences. Specially, Stack AutoEncoder (SAE) is used to build and solve the linear EEG mixing model and the emotion timing model is based on the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). The framework was implemented on the DEAP dataset for an emotion recognition experiment, where the mean accuracy of emotion recognition achieved 81.10% in valence and 74.38% in arousal, and the effectiveness of our framework was verified. Our framework exhibited a better performance in emotion recognition using multi-channel EEG than the compared conventional approaches in the experiments.

Sexual Selection, Agonistic Signaling, and the Effect of Beards on Recognition of Men’s Anger Displays

The beard is arguably one of the most obvious signals of masculinity in humans. Almost 150 years ago, Darwin suggested that beards evolved to communicate formidability to other males, but no studies have investigated whether beards enhance recognition of threatening expressions, such as anger. We found that the presence of a beard increased the speed and accuracy with which participants recognized displays of anger but not happiness (Experiment 1, N = 219). This effect was not due to negative evaluations shared by beardedness and anger or to negative stereotypes associated with beardedness, as beards did not facilitate recognition of another negative expression, sadness (Experiment 2, N = 90), and beards increased the rated prosociality of happy faces in addition to the rated masculinity and aggressiveness of angry faces (Experiment 3, N = 445). A computer-based emotion classifier reproduced the influence of beards on emotion recognition (Experiment 4). The results suggest that beards may alter perceived facial structure, facilitating rapid judgments of anger in ways that conform to evolutionary theory.

Emotion recognition in medical students: effects of facial appearance and care schema activation.

Identifying the factors that may interfere with or sharpen the ability to recognise emotions when observing patients is a critical goal in medical education. This study addressed these issues by investigating the effects of facial appearance bias on medical students' emotion recognition (Experiment 1) and whether such bias is modulated by the activation of relational caregiving schema (Experiment 2). In Experiment 1, medical students were asked to recognise the emotions expressed by individuals differing in facial appearance (trustworthy, neutral and untrustworthy). In Experiment 2, they completed the same type of emotion recognition task after activating and anchoring themselves to the representation of the relational/human competences typical of a competent professional caregiver or after a control non-representation condition. In both experiments, emotion recognition was affected by facial appearance bias: medical students were less accurate and slower in their recognition of emotions displayed by untrustworthy-looking individuals than in their recognition of emotions exhibited by individuals evoking more positive inferences. In Experiment 2, the activation of care schema enhanced medical students' emotion recognition ability regardless of facial appearance-based inferences. Medical students' emotion recognition is affected by appearance-based bias, but such bias may be weakened by techniques that harness medical students' personal affective/relational and representational resources. Thus, the results provide a basis for designing curricula aimed at challenging implicit negative bias and promoting medical students' emotion recognition ability starting in the early stages of their education.

Evaluation of Support Vector Machine and Decision Tree for Emotion Recognition of Malay Folklores

In this paper, the performance of Support Vector Machine (SVM) and Decision Tree (DT) in classifying emotions from Malay folklores is presented. This work is the continuation of our storytelling speech synthesis work to add emotions for a more natural storytelling. A total of 100 documents from children short stories are collected and used as the datasets of the text-based emotion recognition experiment. Term Frequency-Inverse Document Frequency (TF-IDF) is extracted from the text documents and classified using SVM and DT. Four types of common emotions, which are happy, angry, fearful and sad are classified using the two classifiers. Results showed that DT outperformed SVM by more than 22.2% accuracy rate. However, the overall emotion recognition is only at moderate rate suggesting an improvement is needed in future work. The accuracy of the emotion recognition should be improved in future studies by using semantic feature extractors or by incorporating deep learning for classification.

Incongruence Between Observers' and Observed Facial Muscle Activation Reduces Recognition of Emotional Facial Expressions From Video Stimuli.

According to embodied cognition accounts, viewing others’ facial emotion can elicit the respective emotion representation in observers which entails simulations of sensory, motor, and contextual experiences. In line with that, published research found viewing others’ facial emotion to elicit automatic matched facial muscle activation, which was further found to facilitate emotion recognition. Perhaps making congruent facial muscle activity explicit produces an even greater recognition advantage. If there is conflicting sensory information, i.e., incongruent facial muscle activity, this might impede recognition. The effects of actively manipulating facial muscle activity on facial emotion recognition from videos were investigated across three experimental conditions: (a) explicit imitation of viewed facial emotional expressions (stimulus-congruent condition), (b) pen-holding with the lips (stimulus-incongruent condition), and (c) passive viewing (control condition). It was hypothesised that (1) experimental condition (a) and (b) result in greater facial muscle activity than (c), (2) experimental condition (a) increases emotion recognition accuracy from others’ faces compared to (c), (3) experimental condition (b) lowers recognition accuracy for expressions with a salient facial feature in the lower, but not the upper face area, compared to (c). Participants (42 males, 42 females) underwent a facial emotion recognition experiment (ADFES-BIV) while electromyography (EMG) was recorded from five facial muscle sites. The experimental conditions’ order was counter-balanced. Pen-holding caused stimulus-incongruent facial muscle activity for expressions with facial feature saliency in the lower face region, which reduced recognition of lower face region emotions. Explicit imitation caused stimulus-congruent facial muscle activity without modulating recognition. Methodological implications are discussed.

Using a PCA-based dataset similarity measure to improve cross-corpus emotion recognition

In emotion recognition from speech, huge amounts of training material are needed for the development of classification engines. As most current corpora do not supply enough material, a combination of different datasets is advisable. Unfortunately, data recording is done differently and various emotion elicitation and emotion annotation methods are used. Therefore, a combination of corpora is usually not possible without further effort. The manuscript’s aim is to answer the question which corpora are similar enough to jointly be used as training material. A corpus similarity measure based on PCA-ranked features is presented and similar datasets are identified. To evaluate our method we used nine well-known benchmark corpora and automatically identified a sub-set of six most similar datasets. To test that the identified most similar six datasets influence the classification performance, we conducted several cross-corpora emotion recognition experiments comparing our identified six most similar datasets with other combinations. Our most similar sub-set outperforms all other combinations of corpora, the combination of all nine datasets as well as feature normalization techniques. Also influencing side-effects on the recognition rate were excluded. Finally, the predictive power of our measure is shown: increasing similarity score, expressing decreasing similarity, result in decreasing recognition rates. Thus, our similarity measure answers the question which corpora should be included into joint training.

Sparsity constrained differential evolution enabled feature-channel-sample hybrid selection for daily-life EEG emotion recognition

Electroencephalography (EEG) reflects the activities of human brain and it can represent different emotional states to provide impersonal scientific evidence for daily-life emotional health monitoring. However, traditional multi-channel EEG sensing contains irrelevant or even interferential features, channels or samples, leading to redundant data and hardware complexity. This paper proposes a feature-channel-sample hybrid selection method to improve the channel selection, feature extraction and classification scheme for daily-life EEG emotion recognition. The features and channels are selected in pair with sparsity constrained differential evolution where the feature-channel pairs are optimized synchronously in the global search. Furthermore, the distance evaluation is carried out to remove abnormal samples to improve the emotion recognition accuracy. Therefore, efficient feature vectors for valence-arousal classification can be obtained by a small number of sparsely distributed channels. The experiments are based on the widely-used emotion recognition database DEAP and generate a feature-channel-sample hybrid selection scheme with optimized parameter settings. It can be derived that the proposed method can reduce the EEG channels sharply and maintain a relatively high accuracy compared with the related work. Furthermore, by applying this optimal scheme in practice, the real-scene daily-life EEG emotion recognition experiments are carried out on a sparsity constrained web-enabled system and a 10-fold cross validation is organized to confirm the performance. In conclusion, this paper provides a practical and efficient hardware configuration and feature-channel-sample optimal selection scheme for daily-life EEG emotion recognition.