Emotion Recognition Approach Research Articles

CNN based efficient approach for emotion recognition

Determining the psychophysiological state of people has been a significant issue in many fields, such as the adaptation of disabled people to social life. Recently, various physiological signals have been used in emotion recognition studies since the desired success is limited in studies conducted with traditional methods such as posture and facial expressions. In this study, a GoogLeNet-based deep learning method that can automatically detect human emotions using Electroencephalogram (EEG) signals is proposed. EEG signals were converted to EEG images (scalogram) with Continuous Wavelet Transform (CWT), which is more sensitive to Time Frequency (TF) changes in EEG signals. Then, feature extraction was performed from EEG images with a pre-trained GoogLeNet. Finally, the deep features obtained were applied to popular machine learning methods such as k-Nearest Neighbor (k-NN), Support Vector Machine (SVM), and Extreme Learning Machine (ELM) classifiers for emotion classification. The proposed method was tested using the GAMEEMO dataset. This study regards two emotions, namely ‘Positive’ and ‘Negative’ for classification. The comprehensive experimental results show that the proposed method achieved 98.78%, 98.53%, and 98.41% emotion detection accuracy in SVM, k-NN, and ELM classifiers, respectively. The proposed method shows improved performance of about 21.85% than of the existing state-of-the-art methods using the same dataset.

Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion.

Decades of scientific research have been conducted on developing and evaluating methods for automated emotion recognition. With exponentially growing technology, there is a wide range of emerging applications that require emotional state recognition of the user. This paper investigates a robust approach for multimodal emotion recognition during a conversation. Three separate models for audio, video and text modalities are structured and fine-tuned on the MELD. In this paper, a transformer-based crossmodality fusion with the EmbraceNet architecture is employed to estimate the emotion. The proposed multimodal network architecture can achieve up to 65% accuracy, which significantly surpasses any of the unimodal models. We provide multiple evaluation techniques applied to our work to show that our model is robust and can even outperform the state-of-the-art models on the MELD.

Multimodal sentiment and emotion recognition in hyperbolic space

Prior approaches for multimodal sentiment and emotion recognition (SER) exploit input data representations and neural networks based on the classical Euclidean geometry. Recently, however, the hyperbolic metric proved to be a powerful tool for data mapping, being able to capture the hierarchical structure of the relations among elements in the data. In this paper we propose the use of hyperbolic learning for SER, and show that the inclusion in the neural network of hyperbolic structures mapping the input into the hyperbolic space can improve the quality of the predictions. The benefits brought by the hyperbolic features are evaluated by developing extensions of existing methods following two approaches. From one side, we modified state-of-the-art models by including hyperbolic output layers. From the other, we generated hybrid neural network architectures by combining hyperbolic and Euclidean layers according to different schemes. The proposed hyperbolic models were tested on several classification tasks applied to benchmark multimodal SER datasets. Experiments gave strong evidence that in both simple and complex networks the introduction of a hyperbolic structure results in an improvement of the model accuracy. Specifically, the combined use of hyperbolic and Euclidean layers showed superior performance in almost all the classification tasks.

Retraction

This article refers to:RETRACTED ARTICLE: A Robust Face Emotion Recognition Approach through Optimized SIFT Features and Adaptive Deep Belief Neural Network

Correction to: Deep learning approaches for speech emotion recognition: state of the art and research challenges

Correction to: Deep learning approaches for speech emotion recognition: state of the art and research challenges

EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier

Emotion recognition by artificial intelligence (AI) is a challenging task. A wide variety of research has been done, which demonstrated the utility of audio, imagery, and electroencephalography (EEG) data for automatic emotion recognition. This paper presents a new automated emotion recognition framework, which utilizes electroencephalography (EEG) signals. The proposed method is lightweight, and it consists of four major phases, which include: a reprocessing phase, a feature extraction phase, a feature dimension reduction phase, and a classification phase. A discrete wavelet transforms (DWT) based noise reduction method, which is hereby named multi scale principal component analysis (MSPCA), is utilized during the pre-processing phase, where a Symlets-4 filter is utilized for noise reduction. A tunable Q wavelet transform (TQWT) is utilized as feature extractor. Six different statistical methods are used for dimension reduction. In the classification step, rotation forest ensemble (RFE) classifier is utilized with different classification algorithms such as k-Nearest Neighbor (k-NN), support vector machine (SVM), artificial neural network (ANN), random forest (RF), and four different types of the decision tree (DT) algorithms. The proposed framework achieves over 93 % classification accuracy with RFE + SVM. The results clearly show that the proposed TQWT and RFE based emotion recognition framework is an effective approach for emotion recognition using EEG signals.

End-to-End Learning for Multimodal Emotion Recognition in Video With Adaptive Loss

This work presents an approach for emotion recognition in video through the interaction of visual, audio, and language information in an end-to-end learning manner with three key points: 1) lightweight feature extractor, 2) attention strategy, and 3) adaptive loss. We proposed a lightweight deep architecture with approximately 1 MB, which for the most crucial part, accounts for feature extraction, in the emotion recognition systems. The relationship in regard to the time dimension of features is explored with temporal convolutional network instead of RNNs-based architecture to leverage the parallelism and avoid the challenge of vanishing gradient. The attention strategy is employed to adjust the knowledge of temporal networks based on the time dimension and learning of each modality's contribution to the final results. The interaction between the modalities is also investigated when training with adaptive objective function, which adjusts the network's gradient. The experimental results obtained on a large-scale dataset for emotion recognition on Koreans demonstrate the superiority of our method when employing attention mechanism and adaptive loss during training.

Enhanced Facial Emotion Recognition by Optimal Descriptor Selection with Neural Network

Facial Emotion Recognition (FER) is the approach of detecting emotions of humans from facial expressions. Emotions are detected automatically by the human brain and hence, many computer aided techniques are implemented for recognizing emotions. Recognition of natural emotions is an exciting area with a broad range of real time applications like automated tutoring models, smart environments computer–human interaction, driver warning systems, and video and image retrieval. The major goal of this paper is to implement an intelligent model for FER. The proposed model involves few steps: (a) Face extraction (b) Image filtering, (c) Extraction of facial components, (d) Descriptor selection, and (d) Classification. Initially, the face is extracted from the input image by the Viola–Jones method, which is generally employed for the detection of object. Further, the noise of the image is filtered by the Gabor Filtering. Further, the facial components are extracted as features via the Affine-Scale-Invariant Feature Transform (ASIFT), which is the modified version of SIFT method. Since the length of the descriptors generated from ASIFT is high; the number of descriptors is reduced by the optimal descriptor selection approach with the aid of hybrid meta-heuristic algorithm termed as MV-WOA. The extracted descriptors are subjected to Neural Network (NN). As a modification to the existing machine learning algorithms, the number of hidden neurons in the NN is optimized by the proposed WOA+MVO algorithm. By conducting experiments, the results demonstrate that the developed system can perform better the recent traditional approaches for emotion recognition for classifying seven emotions like normal, smile, sad, surprise, anger, fear, and disgust.

Leveraging recent advances in deep learning for audio-Visual emotion recognition

Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation and high-performing deep architectures. The deep feature representations of the audio and visual modalities are fused based on a model-level fusion strategy. A recurrent neural network is then used to capture the temporal dynamics. Our proposed approach substantially outperforms state-of-the-art approaches in predicting valence on the RECOLA dataset. Moreover, our proposed visual facial expression feature extraction network outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets.

Deep learning approaches for speech emotion recognition: state of the art and research challenges

Speech emotion recognition (SER) systems identify emotions from the human voice in the areas of smart healthcare, driving a vehicle, call centers, automatic translation systems, and human-machine interaction. In the classical SER process, discriminative acoustic feature extraction is the most important and challenging step because discriminative features influence the classifier performance and decrease the computational time. Nonetheless, current handcrafted acoustic features suffer from limited capability and accuracy in constructing a SER system for real-time implementation. Therefore, to overcome the limitations of handcrafted features, in recent years, variety of deep learning techniques have been proposed and employed for automatic feature extraction in the field of emotion prediction from speech signals. However, to the best of our knowledge, there is no in-depth review study is available that critically appraises and summarizes the existing deep learning techniques with their strengths and weaknesses for SER. Hence, this study aims to present a comprehensive review of deep learning techniques, uniqueness, benefits and their limitations for SER. Moreover, this review study also presents speech processing techniques, performance measures and publicly available emotional speech databases. Furthermore, this review also discusses the significance of the findings of the primary studies. Finally, it also presents open research issues and challenges that need significant research efforts and enhancements in the field of SER systems.

An Enhanced Emotion Recognition Algorithm Using Pitch Correlogram, Deep Sparse Matrix Representation and Random Forest Classifier

This work presents an approach for text-independent and speaker-independent emotion recognition from speech in real application situations such as noisy and stressful talking conditions. We have incorporated a new way for feature extraction, representation, and noise reduction, replacing the frequently used cepstral features in the literature. The proposed algorithm is modeled as the combination of pitch-correlogram-based noise reduction pre-processing module, sparse-dense decomposition-based feature representation, and random forest classifier. The work is assessed in terms of efficiency and computational complexity using English and Arabic datasets corresponding to noisy and stressful talking conditions. Our system yields significant improvement in results in comparison with other techniques based on the same classifier model. The proposed network architecture achieves significant rise in performance correspond to the recent literature on benchmark datasets.

Data-Driven Model for Emotion Detection in Russian Texts

An important task in the field of automatic data analysis is detecting emotions in texts. The paper presents the approach of emotion recognition for text data in Russian. To conduct an emotion analysis, a method was created based on vector representations of words obtained by the ELMo language model, and subsequent processing by an ensemble classifier. To configure and test the created method, a specially prepared dataset of texts for five basic emotions - joy, sadness, anger, fear, and surprise - is used. The dataset was prepared using a crowdsourcing platform and a home-grown procedure for collecting and controlling annotators’ markup. The overall accuracy is 0.78 (by the F1-macro score), which is currently the new state of the art for Russian. The results can be used for a wide range of tasks, for example: monitoring social moods, generating control signals for mobile robotic systems, etc.

Feasibility and Acceptability of a Mobile-Based Emotion Recognition Approach for Bipolar Disorder

Feasibility and Acceptability of a Mobile-Based Emotion Recognition Approach for Bipolar Disorder

Individual Emotion Recognition Approach Combined Gated Recurrent Unit With Emoticon Distribution Model

With the rapid development of social media, a large increase in text data is occurring in social networks. Mining individual emotions from the data is extremely challenging work in many fields, such as opinion monitoring, business decisions, and information prediction. In previous studies, many studies recognized texts containing positive, negative and neutral sentiments, but few studies focus on emotions according to multiple categories, such as happiness, sadness, anger, fear, disgust and surprise. Sentiment and emotions are two different aspects of individual feelings about matters. In this paper, we propose an individual emotion-identifying model called semantic emoticon emotion recognition (SEER). First, we divide the input short text into four categories by using the emotion dictionary and emoticons. Second, we combine a bidirectional gated recurrent unit (Bi-GRU) network with an attention mechanism to capture the emotion vectors of input text from the word aspect. Third, we construct an emoticon distribution model to obtain the emotion vectors in a large quantity of social network data. Fourth, according to different types of input short texts, we select different fusion weights to fuse the emoticon emotion features in text and the semantic emotion features of the texts. Finally, we classify the short text emotions into six categories depending on the final emotion vector. The experimental results show that the SEER is an effective method for improving the accuracy of emotion recognition. In addition, our proposed approach achieves the highest accuracy of 86.35% in emotion recognition, which indicates that the accuracy of our proposed approach achieves improvements of 12.66%, 14.56%, 5.38%, 4.48%, 4.9%, 2.68%, and 3.17% compared to SVM, WL, LSTM, BiLSTM, GRU, Bi-GRU, and CNN+BiLSTM, respectively.

Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification

Human communication includes rich emotional content, thus the development of multimodal emotion recognition plays an important role in communication between humans and computers. Because of the complex emotional characteristics of a speaker, emotional recognition remains a challenge, particularly in capturing emotional cues across a variety of modalities, such as speech, facial expressions, and language. Audio and visual cues are particularly vital for a human observer in understanding emotions. However, most previous work on emotion recognition has been based solely on linguistic information, which can overlook various forms of nonverbal information. In this paper, we present a new multimodal emotion recognition approach that improves the BERT model for emotion recognition by combining it with heterogeneous features based on language, audio, and visual modalities. Specifically, we improve the BERT model due to the heterogeneous features of the audio and visual modalities. We introduce the Self-Multi-Attention Fusion module, Multi-Attention fusion module, and Video Fusion module, which are attention based multimodal fusion mechanisms using the recently proposed transformer architecture. We explore the optimal ways to combine fine-grained representations of audio and visual features into a common embedding while combining a pre-trained BERT model with modalities for fine-tuning. In our experiment, we evaluate the commonly used CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets for multimodal sentiment analysis. Ablation analysis indicates that the audio and visual components make a significant contribution to the recognition results, suggesting that these modalities contain highly complementary information for sentiment analysis based on video input. Our method shows that we achieve state-of-the-art performance on the CMU-MOSI, CMU-MOSEI, and IEMOCAP dataset.

A Transfer Learning-based Approach for Multimodal Emotion Recognition

The topic of multimodal emotion recognition is one that is expanding at a rapid rate. The goal of this field is to identify and comprehend human emotions through the use of many modalities, such as speech, facial expressions, and physiological data. Transfer learning strategies have been found to be successful in overcoming the issues of processing and integrating material from a variety of modalities, as demonstrated by the findings of a number of studies. For testing multimodal emotion detection models, it is helpful to make use of publicly accessible datasets like IEMOCAP, EmoReact, and AffectNet. They provide useful resources. Data variability, data quality, modality integration, limited labelled data, privacy and ethical issues, and interpretability are only few of the hurdles that must be overcome in order to construct accurate and effective models. In order to address these challenges, a multidisciplinary approach must be taken, and research must continue to be conducted in this area. The goal of this research is to develop more robust and accurate models for multimodal emotion recognition that can be applied across a variety of contexts and populations.

A two-level deep learning approach for emotion recognition in Arabic news headlines

ABSTRACT Going online has created more opportunities for newspapers to present breaking news in a timely manner. Concentrating in spreading more bad news increases the feeling of danger and depression in the society. Some authors believe on tendency of some media to be focused on sharing the bad events in life rather than the good ones because of the impact and the attraction over the audience is more significant. Sentiment analysis work has not been recognized, proposed, or documented on Arabic news because of the challenges that Arabic raises as a language including the different Arabs' dialects and its complex grammatical structure. With the emerging flow of news that cause a global panic and anxiety worldwide, this study focuses on the serious need to identify what effective role could machine learning classifiers have in the early detection process of the psychology impact on the readers by the daily news headlines. In this work, a dataset of news headlines were gathered from top popular online Arabic news sites and were annotated to seven emotional categories: anger, disgust, fear, happiness, neutral, sadness, and surprise. A convolutional neural network-based two-level approach was proposed for sentiment classification. The performance of the proposed approach was compared to six machine learning classifiers (zeroR, k-nearest neighbor, decision trees, naïve Bayes, random forest and Support vector machine) and showed better accuracy, precision, and recall.

Optimal feature selection for speech emotion recognition using enhanced cat swarm optimization algorithm

Human interactions involve emotional cues that can be used to interpret the emotion expressed by the speaker. As the vocal emotions vary from one speaker to another, there is a chance of misinterpretation. To determine the emotion expressed by the speaker, a speech emotion recognizer can be utilized. It is known that speech expresses the emotional states of humans along with the syntax and semantic content of linguistic sentences. Therefore, human emotion recognition using speech signaling is possible. Speech emotion recognition is a crucial and challenging task in which the feature extraction plays a prominent role in its performance. Determining emotional states in speech signals is a very challenging area for many reasons. The first issue of all speech emotion systems is the selection of the best features, which is powerful enough to distinguish various emotions. The presence of different language, pronunciation, sentences, style, and speakers adds additional difficulty since these characteristics include pitch and energy that directly alters most of the features extracted. Redundant features and high computational cost make emotion recognition an undesirable task. Instead of focusing on the words, the vocal changes and communicative pressure on the words should be taken as the primary consideration. The Enhanced Cat Swarm Optimization (ECSO) algorithm for feature extraction has been proposed to address these issues and it is not used in any existing speech emotion recognition approaches. The proposed approach achieves excellent performance in terms of accuracy, recognition rate, sensitivity, and specificity.

Music emotion recognition using convolutional long short term memory deep neural networks

In this paper, we propose an approach for music emotion recognition based on convolutional long short term memory deep neural network (CLDNN) architecture. In addition, we construct a new Turkish emotional music database composed of 124 Turkish traditional music excerpts with a duration of 30 s each and the performance of the proposed approach is evaluated on the constructed database. We utilize features obtained by feeding convolutional neural network (CNN) layers with log-mel filterbank energies and mel frequency cepstral coefficients (MFCCs) in addition to standard acoustic features. Classification results show that the best performance is obtained when the new feature set is combined with the standard features using the long short term memory (LSTM) + deep neural network (DNN) classi fier. The overall accuracy of 99.19% is obtained using the proposed system with 10 fold cross-validation. Specifically, 6.45 points improvement is achieved. Additionally, the results also show that the LSTM + DNN classifier yields 1.61, 1.61 and 3.23 points improvements in music emotion recognition accuracies compared to k-nearest neighbor (k-NN), support vector machine (SVM), and Random Forest classifiers, respectively.

Emotion Sensing From Head Motion Capture

Computational analysis of emotion from verbal and non-verbal behavioral cues is critical for human-centric intelligent systems. Among the non-verbal cues, head motion has received relatively less attention, although its importance has been noted in several research. We propose a new approach for emotion recognition using head motion captured using Motion Capture (MoCap). Our approach is motivated by the well known kinesics-phonetic analogy , which advocates that, analogous to human speech being composed of phonemes, head motion is composed of kinemes i.e., elementary motion units. We discover a set of kinemes from head motion in an unsupervised manner by projecting them onto a learned basis domain and subsequently clustering them. This transforms any head motion to a sequence of kinemes. Next, we learn the temporal latent structures within the kineme sequence pertaining to each emotion. For this purpose, we explore two separate approaches – one using Hidden Markov Model and another using neural network. This class-specific, kineme-based representation of head motion is used to perform emotion recognition on the popular IEMOCAP database. We achieve high recognition accuracy (61.8% for three class) for various emotion recognition tasks using head motion alone . This work adds to our understanding of head motion dynamics, and has applications in emotion analysis and head motion animation and synthesis.