Automatic Emotion Recognition System Research Articles

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Automatic emotion recognition is the process of identifying human emotion from signals such as facial expression, speech, and text. Collecting and labelling such signals is often tedious and many times requires expert knowledge. An effective way to address challenges related to the scarcity of data and lack of human labels, is transfer learning. In this manuscript, we will describe fundamental concepts in the field of transfer learning and review work which has successfully applied transfer learning for automatic emotion recognition. We will finally discuss promising future research directions of transfer learning for improving the generalizability of automatic emotion recognition systems.

Audio compression with multi-algorithm fusion and its impact in speech emotion recognition

The study examines the impact of multi-algorithm fusion over audio compression with reference to the traditional exercises. For emotion recognition, here the most prominent features ‘Mel Frequency Cepstral Coefficients’ (MFCC) and ‘Discrete Wavelet Transform’ (DWT) features are extracted from prevalent speech samples of Berlin emotional database and Telugu (a south Indian language) database, we proposed automatic emotion recognition system (AERS) based on multi-algorithms fusion. AERS means to monitor and identify unit psychological/emotional state. The extracted features are analyzed using support vector machine, K-NN algorithms used for the classification of different states of emotion. Using two state-of-art mp3, Speex codec with different bit-rates investigated to ensure specific emotional intelligibility. MP3 codec configuration with 96 kbps bit-rate is recommended to pull off high compression for all emotions. Fusion algorithms also performed well compared with individual algorithms. Accuracy of 94.2% using fusion DWT and MFCC compared to 89.1% using DWT and 91.38% using MFCC separately. The accuracy of the proposed method increased further to 94% through a multiresolution approach by approximating frequency along with time information.

Efficient speech emotion recognition using modified feature extraction

Abstract One of the most important issues in human-computer interaction is to create a system that can hear and respond correctly like a human. This has led to the design of the Automatic Speech Emotion Recognition system (SER) that is able to identify different emotional classes by extracting and selecting effective features from speech signals. For this reason, in this study, we propose a novel feature extraction method based on adaptive time-frequency coefficients to improve the SER. The simulations are performed using the Berlin Emotional Speech Database (EMO-DB), the Surrey Audio-Visual Expressed Emotion Database (SAVEE), and the Persian Drama Radio Emotional Corpus (PDREC). The main contribution of our work is to extract novel features, called Adaptive Time-Frequency features, based on the Fractional Fourier Transform and to combine them with Cepstral features. Experimental results show that the proposed method effectively identifies different emotional classes in EMO-DB (97.57% accuracy), SAVEE (80% accuracy), and PDREC (91.46% accuracy) data sets.

The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features.

Recently, developing an accurate automatic emotion recognition system using a minimum number of bio-signals has become a challenging issue in "affective computing." This study aimed to propose a reliable system by examining nonlinear dynamics of photoplethysmogram (PPG) and galvanic skin response (GSR). To address this goal, two strategies were adopted. First, the efficiency of each signal in valence/arousal based emotion categorization was examined. Then, the proficiency of a hybrid feature, by combining both GSR and PPG features was studied. Lyapunov exponents, lagged Poincare's measures, and approximate entropy were extracted to characterize the irregularity and chaotic behavior of the phase space. To discriminate two levels of arousal and two levels of the valence, a probabilistic neural network (PNN) with different sigma adjustment parameter was examined. The results showed that the phase space geometry and consequently, the signal dynamics are influenced by the emotional music video. Additionally, distinctive patterns of the phase space behavior were observed under the influence of different lags. For both signals, the most irregularity was observed during the high valence, and the least irregularity was seen during the low valence. Consequently, signals' irregularity is affected by the valence dimension. The results showed that the fusion has more potential for emotion recognition than that of using each signal separately. For sigma = 0.1, the highest recognition rate was 100% in a subject-dependent mode. In a subject-independent mode, the maximum accuracies of 88.57 and 86.8% were obtained for arousal and valence dimensions, respectively.

Towards emotion recognition from contextual information using machine learning

Emotions influence cognitive processes that underlie human behavior. Whereas experiencing negative emotions may lead to develop psychological disorders, experiencing positive emotions may improve creative thinking and promote cooperative behavior. The importance of human emotions has led to the development of automatic emotion recognition systems based on analysis of speech waveforms, facial expressions, and physiological signals as well as text data mining. However, emotions are associated with a context (in which emotions are actually experienced), hence, this work focuses on emotion recognition from contextual information. In this paper, we present a study aimed to assess the feasibility of automatically recognizing emotions from individuals’ contexts. In this study, 32 participants provided information using a mobile application about their emotions and the context (e.g., companions, activities, and locations) in which these emotions were experienced. We used machine learning techniques to build individual models, general models, and gender-specific models to automatically recognize emotions of participants. The empirical results show that individuals’ emotions are highly related to their context and that automatic recognition of emotions in real-world situations is feasible by using contextual data.

Multi-modal Correlated Network for emotion recognition in speech

With the growing demand of automatic emotion recognition system, emotion recognition is becoming more and more crucial for human–computer interaction (HCI) research. Recently, there is a continuous improvement in the performance of automatic emotion recognition due to the development of both hardware and deep learning methods. However, because of the abstract concept and multiple expressions of emotion, automatic emotion recognition is still a challenging task. In this paper, we propose a novel Multi-modal Correlated Network for emotion recognition aiming at exploiting the information from both audio and visual channels to achieve more robust and accurate detection. In the proposed method, the audio signals and visual signals are first preprocessed for the feature extraction. After preprocessing, we obtain the Mel-spectrograms, which can be treated as images, and the representative frames from visual segments. Then the Mel-spectrograms are fed to the convolutional neural network (CNN) to get the audio features and the representative frames are fed to the CNN and LSTM to get features. Specially, we employ the triplet loss to increase the differentiation of inter-class. Meanwhile, we propose a novel correlated loss to reduce the differentiation of intra-class. Finally, we apply the feature fusion method to fuse the audio and visual feature for emotion recognition classification. The experimental result on AEFW dataset demonstrates the correlation information of multiple modals is crucial for automatic emotion recognition and the proposed method can achieve the state-of-the-art performance on the classification task.

Emotion recognition using secure edge and cloud computing

This paper proposes an edge-cloud based privacy-preserved automatic emotion recognition system in which Internet of Things (IoT) devices capture facial image and speech signals from a user before distribution to different edge clouds using a secret sharing scheme, ensuring the privacy of the user. Simple signal preprocessing is performed in the edge clouds and then the signals are transmitted to the core cloud. In the core cloud, a Convolutional Neural Network (CNN)-based, pre-trained model is applied to extract deep-learned features from image and speech signals. Following the CNN model, two deep sparse auto-encoders fuse the signal features and introduce an extended non-linearity to the features. Finally, a support vector machine classifies the input in a corresponding emotion class. Several experiments were performed using the proposed system on two publicly available databases, namely, the RML database and eNTERFACE’05 database. The highest recognition accuracies by the proposed system on the two databases were 82.3% and 87.6%, respectively. These accuracies are higher than those by other state-of-the-art systems.

An Audio-Visual Emotion Recognition System Using Deep Learning Fusion for a Cognitive Wireless Framework

Automatically recognizing emotions of patients can be a good facilitator of a connected healthcare framework. It can give automatic feedback to the stakeholders of the healthcare industry about patients' states and satisfaction levels. In this article, we propose an automatic audio-visual emotion recognition system in a connected healthcare framework. The system uses a 2D CNN model for the speech modality and a 3D CNN model for the visual modality. For the speech signal, preprocessing is done to extract the PS-PA feature vector. The features from the two CNN models are blended by two ELM networks. The first ELM is trained with gender-specific data, while the other one is trained with emotion-specific data. The proposed system is evaluated using three databases, and the experiments prove the success of the system. In the healthcare framework, we use edge computing prior to intensive-processing cloud computing. In the edge computing, we realize edge caching, which can store the CNN model parameters and thereby perform the testing fast.

A novel approach to emotion recognition using local subset feature selection and modified Dempster-Shafer theory

BackgroundEmotion recognition is an increasingly important field of research in brain computer interactions.IntroductionWith the advance of technology, automatic emotion recognition systems no longer seem far-fetched. Be that as it may, detecting neural correlates of emotion has remained a substantial bottleneck. Settling this issue will be a breakthrough of significance in the literature.MethodsThe current study aims to identify the correlations between different emotions and brain regions with the help of suitable electrodes. Initially, independent component analysis algorithm is employed to remove artifacts and extract the independent components. The informative channels are then selected based on the thresholded average activity value for obtained components. Afterwards, effective features are extracted from selected channels common between all emotion classes. Features are reduced using the local subset feature selection method and then fed to a new classification model using modified Dempster-Shafer theory of evidence.ResultsThe presented method is employed to DEAP dataset and the results are compared to those of previous studies, which highlights the significant ability of this method to recognize emotions through electroencephalography, by the accuracy of about 91%. Finally, the obtained results are discussed and new aspects are introduced.ConclusionsThe present study addresses the long-standing challenge of finding neural correlates between human emotions and the activated brain regions. Also, we managed to solve uncertainty problem in emotion classification which is one of the most challenging issues in this field. The proposed method could be employed in other practical applications in future.

Prediction of Emotion Change From Speech

The fact that emotions are dynamic in nature and evolve across time has been explored relatively less often in automatic emotion recognition systems to date. Although within-utterance information about emotion changes recently has received some attention, there remain open questions unresolved, such as how to approach delta emotion ground truth, how to predict the extent of emotion change from speech, and how well change can be predicted relative to absolute emotion ratings. In this article, we investigate speech-based automatic systems for continuous prediction of the extent of emotion changes in arousal/valence. We propose the use of regression (smoothed) deltas as ground truth for emotion change, which yielded considerably higher inter-rater reliability than first-order deltas, a commonly used approach in previous research, and represent a more appropriate approach to derive annotations for emotion change research, findings which are applicable beyond speech-based systems. In addition, the first system design for continuous emotion change prediction from speech is explored. Experimental results under the Output-Associative Relevance Vector Machine framework interestingly show that changes in emotion ratings may be better predicted than absolute emotion ratings on the RECOLA database, achieving 0.74 vs 0.71 for arousal and 0.41 vs 0.37 for valence in concordance correlation coefficients. However, further work is needed to achieve effective emotion change prediction performances on the SEMAINE database, due to the large number of non-change frames in the absolute emotion ratings.

Speech Emotion Classification and Recognition with different methods for Turkish Language

In several
 application, emotion  recognition from
 the speech signal has been research topic since many years. To determine the
 emotions from the speech signal, many systems have been developed. To solve the
 speaker emotion recognition problem, hybrid model is proposed to classify five
 speech emotions, including  anger,
 sadness, fear, happiness and neutral. The aim this study of was to actualize
 automatic voice and speech emotion recognition system using hybrid model taking
 Turkish sound forms and properties into consideration.  Approximately 3000 Turkish voice samples of
 words and clauses with differing lengths have been collected from 25 males
 and  25 females. In this study, an
 authentic and unique  Turkish  database has been used. Features of these
 voice samples have been obtained using Mel Frequency Cepstral Coefficients
 (MFCC) and Mel Frequency Discrete Wavelet Coefficients (MFDWC). Moreover,
 spectral features of these voice samples have been obtained  using Support Vector Machine (SVM). Feature
 vectors of the voice samples obtained have been trained with such methods as
 Gauss Mixture Model( GMM), Artifical Neural Network (ANN), Dynamic Time Warping
 (DTW), Hidden Markov Model (HMM) and hybrid model(GMM with combined SVM).  This hybrid model has been carried out by
 combining with SVM and GMM.  In first
 stage of this model, with SVM has been performed  subsets obtained vector of  spectral features. In the second  phase, a set of training and tests have been
 formed from these spectral features. In the test phase, owner of a given voice
 sample has been identified taking the trained voice samples into consideration.
 Results and performances of the algorithms employed in the study for
 classification have been also demonstrated in a comparative manner.         

Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots.

Designing an efficient automatic emotion recognition system based on physiological signals has attracted great interests within the research of human-machine interactions. This study was aimed to classify emotional responses by means of a simple dynamic signal processing technique and fusion frameworks. The electrocardiogram and finger pulse activity of 35 participants were recorded during rest condition and when subjects were listening to music intended to stimulate certain emotions. Four emotion categories, including happiness, sadness, peacefulness, and fear were chosen. Estimating heart rate variability (HRV) and pulse rate variability (PRV), 4 Poincare indices in 10 lags were extracted. The support vector machine classifier was used for emotion classification. Both feature level (FL) and decision level (DL) fusion schemes were examined. Significant differences have been observed between lag 1 Poincare plot indices and the other lagged measures. The mean accuracies of 84.1, 82.9, 79.68, and 76.05% were obtained for PRV, DL, FL, and HRV measures, respectively. However, DL outperformed others in discriminating sadness and peacefulness, using SD1 and total features, correspondingly. In both cases, the classification rates improved up to 92% (with the sensitivity of 95% and specificity of 83.33%). Totally, DL resulted in better performances compared to FL. In addition, the impact of the fusion rules on the classification performances has been confirmed.

Cloud-oriented emotion feedback-based Exergames framework

Exergames platform can be more appealing to the users if they can interact with emotion. Therefore, in this paper, we propose an automatic emotion recognition system from speech to be embedded in the Exergames platform. While playing and doing exercise, the user expresses his or her feeling by uttering some phrases. The speech is recorded by an omnidirectional mic and transmitted to an emotion recognition server in a cloud environment, where the emotion (e.g. happy, sad or neutral) is recognized. For the recognition, we use MPEG-7 low-level audio features and a Gaussian mixture model based classifier. A tactile vibration is generated based on the emotion and feedback to the user for a real feeling. The user can thus have an instantaneous vibrational feeling based on his or her satisfaction. The recognized emotion can also be used as the user’s satisfaction of the framework on the fly without the need of a survey session. The experimental study and performance comparison show that the proposed framework has positive effects on the perception of physical activities.

Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Data

Emotion recognition is an important field of research in Brain Computer Interactions. As technology and the understanding of emotions are advancing, there are growing opportunities for automatic emotion recognition systems. Neural networks are a family of statistical learning models inspired by biological neural networks and are used to estimate functions that can depend on a large number of inputs that are generally unknown. In this paper we seek to use this effectiveness of Neural Networks to classify user emotions using EEG signals from the DEAP (Koelstra et al (2012)) dataset which represents the benchmark for Emotion classification research. We explore 2 different Neural Models, a simple Deep Neural Network and a Convolutional Neural Network for classification. Our model provides the state-of-the-art classification accuracy, obtaining 4.51 and 4.96 percentage point improvements over (Rozgic et al (2013)) classification of Valence and Arousal into 2 classes (High and Low) and 13.39 and 6.58 percentage point improvements over (Chung and Yoon(2012)) classification of Valence and Arousal into 3 classes (High, Normal and Low). Moreover our research is a testament that Neural Networks could be robust classifiers for brain signals, even outperforming traditional learning techniques.

Performance Analysis of Malayalam Language Speech Emotion Recognition System Using ANN/SVM

Automatic recognition of emotions from speech by machines has been one of the most challenging areas of research in the field of human machine interaction. Automatic emotion recognition system by speech merely means that to monitor and identify the emotional or physiological state of an individual from their utterances. Speech emotion recognition has wide range of application ranging from clinical studies to robotics. In this paper developed speech emotional database for Malayalam language (One of the south Indian languages) and a system for recognizing the emotions. The system used Mel Frequency Cepstral Coefficients (MFCCs), Short Time Energy (STE) and Pitch as features extraction techniques. Two classifiers, namely Artificial Neural Network (ANN) and Support Vector Machine (SVM) used for pattern classification. Experiments show that this method provides a high accuracy of 88.4% in the case of ANN and 78.2% in the case of SVM.

Razvoj zbirke slovenskega čustvenega govora iz radijskih iger – EmoLUKS

V prispevku predstavljamo graditev slovenske zbirke čustvenega govora za umetno tvorjenje govora in hkrati raziščemo tudi možnosti njene uporabe pri razpoznavanju čustvenega stanja govorca. V prispevku se osredotočamo na opis razvite metodologije za označevanje paralingvistične informacije v govoru na primeru označevanja čustvenih stanj v slovenskih radijskih igrah. Zbirka vsebuje govorne zvočne signale sedemnajstih radijskih iger. Trenutno označeno gradivo obsega čustven govor enega govorca in ene govorke. Čustvene oznake posnetkov smo pridobili s pomočjo dvostopenjskega označevanja s petimi prostovoljnimi označevalci, ki so označili posnetke v dveh časovno ločenih intervalih. Način označevanja omogoča medsebojno primerjavo oznak označevalcev. S pomočjo označenega gradiva v obeh iteracijah poročamo o konsistentnosti označevalcev in ujemanju njihovih mnenj. Na podlagi večinskega mnenja pridobljenih čustvenih oznak vsakemu posnetku pripišemo tisto čustveno oznako, ki je bila med označevalci največkrat izbrana, in tako označene posnetke združimo v zbirko čustvenega govora EmoLUKS, ki jo kvantitativno in kvalitativno ovrednotimo z uporabo uveljavljenega samodejnega sistema za razpoznavanje čustvenih stanj govorca. Konsistentnost oznak ovrednotimo z dvorazrednim in sedemrazrednim od govorca odvisnim razvrščevalnikom čustvenih stanj. Uspešni rezultati razpoznavanja dodatno potrjujejo, da podatkovna zbirka kljub svoji zahtevnosti vsebuje jasno izražena čustvena stanja govorca.

Multimodal emotion recognition based on peak frame selection from video

We present a fully automatic multimodal emotion recognition system based on three novel peak frame selection approaches using the video channel. Selection of peak frames (i.e., apex frames) is an important preprocessing step for facial expression recognition as they contain the most relevant information for classification. Two of the three proposed peak frame selection methods (i.e., MAXDIST and DEND-CLUSTER) do not employ any training or prior learning. The third method proposed for peak frame selection (i.e., EIFS) is based on measuring the “distance” of the expressive face from the subspace of neutral facial expression, which requires a prior learning step to model the subspace of neutral face shapes. The audio and video modalities are fused at the decision level. The subject-independent audio-visual emotion recognition system has shown promising results on two databases in two different languages (eNTERFACE and BAUM-1a).

New approach in quantification of emotional intensity from the speech signal: emotional temperature

The automatic speech emotion recognition has a huge potential in applications of fields such as psychology, psychiatry and the affective computing technology. The spontaneous speech is continuous, where the emotions are expressed in certain moments of the dialogue, given emotional turns. Therefore, it is necessary that the real-time applications are capable of detecting changes in the speaker's affective state. In this paper, we emphasize on recognizing activation from speech using a few feature set obtained from a temporal segmentation of the speech signal of different language like German, English and Polish. The feature set includes two prosodic features and four paralinguistic features related to the pitch and spectral energy balance. This segmentation and feature set are suitable for real-time emotion applications because they allow detect changes in the emotional state with very low processing times. The German Corpus EMO-DB (Berlin Database of Emotional Speech), the English Corpus LDC (Emotional Prosody Speech and Transcripts database) and the Polish Emotional Speech Database are used to train the Support Vector Machine (SVM) classifier and for gender-dependent activation recognition. The results are analyzed for each speech emotion with gender-dependent separately and obtained accuracies of 94.9%, 88.32% and 90% for EMO-DB, LDC and Polish databases respectively. This new approach provides a comparable performance with lower complexity than other approaches for real-time applications, thus making it an appealing alternative, may assist in the future development of automatic speech emotion recognition systems with continuous tracking.

Feature Selection for GUMI Kernel-Based SVM in Speech Emotion Recognition

Speech emotion recognition is the indispensable requirement for efficient human machine interaction. Most modern automatic speech emotion recognition systems use Gaussian mixture models (GMM) and Support Vector Machines (SVM). GMM are known for their performance and scalability in the spectral modeling while SVM are known for their discriminatory power. A GMM-supervector characterizes an emotional style by the GMM parameters (mean vectors, covariance matrices, and mixture weights). GMM-supervector SVM benefits from both GMM and SVM frameworks. In this paper, the GMM-UBM mean interval (GUMI) kernel based on the Bhattacharyya distance is successfully used. CFSSubsetEval combined with Best first algorithm and Greedy stepwise were also utilized on the supervectors space in order to select the most important features. This framework is illustrated using Mel-frequency cepstral (MFCC) coefficients and Perceptual Linear Prediction (PLP) features on two different emotional databases namely the Surrey Audio-Expressed Emotion and the Berlin Emotional speech Database.

Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

Automatic emotion recognition systems based on supervised machine learning require reliable annotation of affective behaviours to build useful models. Whereas the dimensional approach is getting more and more popular for rating affective behaviours in continuous time domains, e.g., arousal and valence, methodologies to take into account reaction lags of the human raters are still rare. We therefore investigate the relevance of using machine learning algorithms able to integrate contextual information in the modelling, like long short-term memory recurrent neural networks do, to automatically predict emotion from several (asynchronous) raters in continuous time domains, i.e., arousal and valence. Evaluations are performed on the recently proposed RECOLA multimodal database (27 subjects, 5 min of data and six raters for each), which includes audio, video, and physiological (ECG, EDA) data. In fact, studies uniting audiovisual and physiological information are still very rare. Features are extracted with various window sizes for each modality and performance for the automatic emotion prediction is compared for both different architectures of neural networks and fusion approaches (feature-level/decision-level). The results show that: (i) LSTM network can deal with (asynchronous) dependencies found between continuous ratings of emotion with video data, (ii) the prediction of the emotional valence requires longer analysis window than for arousal and (iii) a decision-level fusion leads to better performance than a feature-level fusion. The best performance (concordance correlation coefficient) for the multimodal emotion prediction is 0.804 for arousal and 0.528 for valence.