Automatic Emotion Recognition System Research Articles

A Survey on Speech Emotion Recognition System using CNN Algorithm

The speech emotion recognition system (SER) plays an important role in decoding and predicting the speaker's emotional state by analyzing audio signals. Emotions are often simplified and grouped into categories such as anger, happiness, sadness, and even neutral emotional states. These emotions serve to communicate one's thoughts and provide insight into one's physical and mental health. Speech is the most basic and natural form of communication when interacting with others. Voice signals enable rapid communication between individuals, making them a valuable and effective method of expression. Over the past decades, countless research initiatives have been devoted to the development of voice-based automatic emotion recognition systems, especially to improve human-machine communication. Voice is gradually taking center stage in the field of Human-Machine interfaces in the IT sector. This interdisciplinary field draws on a variety of fields, including computer science, signal processing, psychology, linguistics, and more. As technology advances, it allows for seamless communication between humans and machines. Speech recognition not only interprets speech but also captures nuances in a person's tone and expressions, similar to body language. Therefore, it becomes an essential element of the Human-Machine communication system.

Emotion recognition by skeleton-based spatial and temporal analysis

This study introduces an automatic emotion recognition system (AER) focusing on skeletal-based kinematic datasets for enhanced human–computer interaction. Departing from conventional approaches, it achieves real-time emotion recognition in real-life situations. The dataset covers seven emotions and undergoes assessment by eight diverse machine and deep learning algorithms. A thorough investigation is undertaken by varying window sizes and data states, including raw positions and feature-extracted data. The findings imply that incorporating advanced techniques like joint-related feature extraction and robust classifier models yields promising outcomes. Dataset augmentation via varying window sizes enriches insights into real-world scenarios. Evaluations exhibit classification accuracy surpassing 99% for small windows, 94% for medium, and exceeding 88% for larger windows, thereby confirming the robust nature of the approach. Furthermore, we highlight window size's impact on emotion detection and the benefits of combining coordinate axes for efficiency and accuracy. The analysis intricately examines the contributions of features at both the joint and axis levels, assisting in making well-informed selections. The study's contributions include carefully curated datasets, transparent code, and models, all of which ensure the possibility of replication. The paper establishes a benchmark that bridges theory and practicality, solidifying the proposed approach's effectiveness in balancing accuracy and efficiency. By pioneering advanced AER through kinematic data, it sets a new standard for efficacy while driving seamless human–computer interaction through rigorous analysis and strategic design.

Lemniscate of Bernoulli's map quantifiers: innovative measures for EEG emotion recognition.

Thanks to the advent of affective computing, designing an automatic human emotion recognition system for clinical and non-clinical applications has attracted the attention of many researchers. Currently, multi-channel electroencephalogram (EEG)-based emotion recognition is a fundamental but challenging issue. This experiment envisioned developing a new scheme for automated EEG affect recognition. An innovative nonlinear feature engineering approach was presented based on Lemniscate of Bernoulli's Map (LBM), which belongs to the family of chaotic maps, in line with the EEG's nonlinear nature. As far as the authors know, LBM has not been utilized for biological signal analysis. Next, the map was characterized using several graphical indices. The feature vector was imposed on the feature selection algorithm while evaluating the role of the feature vector dimension on emotion recognition rates. Finally, the efficiency of the features on emotion recognition was appraised using two conventional classifiers and validated using the Database for Emotion Analysis using Physiological signals (DEAP) and SJTU Emotion EEG Dataset-IV (SEED-IV) benchmark databases. The experimental results showed a maximum accuracy of 92.16% for DEAP and 90.7% for SEED-IV. Achieving higher recognition rates compared to the state-of-art EEG emotion recognition systems suggest the proposed method based on LBM could have potential both in characterizing bio-signal dynamics and detecting affect-deficit disorders.

Automatic Emotion Recognition in Clinical Scenario: A Systematic Review of Methods

Automatic emotion recognition has powerful opportunities in the clinical field, but several critical aspects are still open, such as heterogeneity of methodologies or technologies tested mainly on healthy people. This systematic review aims to survey automatic emotion recognition systems applied in real clinical contexts, to deeply analyse clinical and technical aspects, how they were addressed, and relationships among them. The literature review was conducted on: IEEEXplore, ScienceDirect, Scopus, PubMed, ACM. Inclusion criteria were the presence of an automatic emotion recognition algorithm and the enrollment of at least 2 patients in the experimental protocol. The review process followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Moreover, the works were analysed according to a reference model to deeply examine both clinical and technical topics. 52 scientific papers passed inclusion criteria. Most clinical scenarios involved neurodevelopmental, neurological and psychiatric disorders with the aims of diagnosing, monitoring, or treating emotional symptoms. The most adopted signals are video and audio, while supervised shallow learning is mostly used for emotion recognition. A poor study design, tiny samples, and the absence of a control group emerged as methodological weaknesses. Heterogeneity of performance metrics, datasets and algorithms challenges results comparability, robustness, reliability and reproducibility.

Task-specific speech enhancement and data augmentation for improved multimodal emotion recognition under noisy conditions

Automatic emotion recognition (AER) systems are burgeoning and systems based on either audio, video, text, or physiological signals have emerged. Multimodal systems, in turn, have shown to improve overall AER accuracy and to also provide some robustness against artifacts and missing data. Collecting multiple signal modalities, however, can be very intrusive, time consuming, and expensive. Recent advances in deep learning based speech-to-text and natural language processing systems, however, have enabled the development of reliable multimodal systems based on speech and text while only requiring the collection of audio data. Audio data, however, is extremely sensitive to environmental disturbances, such as additive noise, thus faces some challenges when deployed “in the wild.” To overcome this issue, speech enhancement algorithms have been deployed at the input signal level to improve testing accuracy in noisy conditions. Speech enhancement algorithms can come in different flavors and can be optimized for different tasks (e.g., for human perception vs. machine performance). Data augmentation, in turn, has also been deployed at the model level during training time to improve accuracy in noisy testing conditions. In this paper, we explore the combination of task-specific speech enhancement and data augmentation as a strategy to improve overall multimodal emotion recognition in noisy conditions. We show that AER accuracy under noisy conditions can be improved to levels close to those seen in clean conditions. When compared against a system without speech enhancement or data augmentation, an increase in AER accuracy of 40% was seen in a cross-corpus test, thus showing promising results for “in the wild” AER.

Development of a speech emotion recognizer for large-scale child-centered audio recordings from a hospital environment

In order to study how early emotional experiences shape infant development, one approach is to analyze the emotional content of speech heard by infants, as captured by child-centered daylong recordings, and as analyzed by automatic speech emotion recognition (SER) systems. However, since large-scale daylong audio is initially unannotated and differs from typical speech corpora from controlled environments, there are no existing in-domain SER systems for the task. Based on existing literature, it is also unclear what is the best approach to deploy a SER system for a new domain. Consequently, in this study, we investigated alternative strategies for deploying a SER system for large-scale child-centered audio recordings from a neonatal hospital environment, comparing cross-corpus generalization, active learning (AL), and domain adaptation (DA) methods in the process. We first conducted simulations with existing emotion-labeled speech corpora to find the best strategy for SER system deployment. We then tested how the findings generalize to our new initially unannotated dataset. As a result, we found that the studied AL method provided overall the most consistent results, being less dependent on the specifics of the training corpora or speech features compared to the alternative methods. However, in situations without the possibility to annotate data, unsupervised DA proved to be the best approach. We also observed that deployment of a SER system for real-world daylong child-centered audio recordings achieved a SER performance level comparable to those reported in literature, and that the amount of human effort required for the system deployment was overall relatively modest.

Emotion Recognition System Based on Two-Level Ensemble of Deep-Convolutional Neural Network Models

Emotions play a crucial role in human interaction and healthcare. This study introduces an automatic emotion recognition system based on deep learning using electroencephalogram signals. A lightweight pyramidal one-dimensional convolutional neural network model is proposed that involves a small number of learnable parameters. Using the model, a two-level ensemble classifier is designed. Each channel is scanned incrementally in the first level to generate predictions, which are fused using the majority vote. The second level fuses the predictions of all the channels of a signal using a majority vote to predict the emotional state. The method was validated using the public domain challenging benchmark dataset. The electroencephalogram signals over five brain regions were analyzed. The results indicate that the frontal brain region plays a dominant role in two emotion recognition problems (distinguishing high valence vs. low valence and high arousal vs. low arousal states).

An efficient facial expression recognition system with appearance-based fused descriptors

Facial expressions have always been defined as the most efficient way to characterize emotions. Therefore, they have been used in various fields of applications such as human-computer interaction. With the continuous progress in terms of image processing and machine learning techniques, several new methods are proposed each year. However, their performance still needs to be improved and this task remains challenging. In this paper, we propose an automatic emotion recognition system through facial analysis. We compare the use of two distinct image descriptors namely, Local Binary Pattern and Histogram of Oriented Gradients with two different dimensionality reduction techniques namely, Principal Component Analysis and Locally Linear Embedding. Moreover, we also combine both descriptors in order to improve the system efficiency. For the classification part of the system, we choose to use the multi-class Support Vector Machine classifier for its generalization capabilities to distinguish between the six basic emotions. Finally, we assess the performance of the proposed system using three different benchmark facial expression datasets namely, JApanese Female Facial Expression (JAFFE), Karolinska Directed Emotional Faces (KDEF) and Radboud Faces Database (RaFD) which yield promising recognition rates with 97.16%, 90.12% and 95.54%, respectively.

ASERNet: Automatic speech emotion recognition system using MFCC-based LPC approach with deep learning CNN

Automatic speech emotion recognition (ASER) from source speech signals is quite a challenging task since the recognition accuracy is highly dependent on extracted features of speech that are utilized for the classification of speech emotion. In addition, pre-processing and classification phases also play a key role in improving the accuracy of ASER system. Therefore, this paper proposes a deep learning convolutional neural network (DLCNN)-based ASER model, hereafter denoted with ASERNet. In addition, the speech denoising is employed with spectral subtraction (SS) and the extraction of deep features is done using integration of linear predictive coding (LPC) with Mel-frequency Cepstrum coefficients (MFCCs). Finally, DLCNN is employed to classify the emotion of speech from extracted deep features using LPC-MFCC. The simulation results demonstrate the superior performance of the proposed ASERNet model in terms of quality metrics such as accuracy, precision, recall, and F1-score, respectively, compared to state-of-the-art ASER approaches.

Smart Classroom Monitoring Using Novel Real-Time Facial Expression Recognition System

Emotions play a vital role in education. Technological advancement in computer vision using deep learning models has improved automatic emotion recognition. In this study, a real-time automatic emotion recognition system is developed incorporating novel salient facial features for classroom assessment using a deep learning model. The proposed novel facial features for each emotion are initially detected using HOG for face recognition, and automatic emotion recognition is then performed by training a convolutional neural network (CNN) that takes real-time input from a camera deployed in the classroom. The proposed emotion recognition system will analyze the facial expressions of each student during learning. The selected emotional states are happiness, sadness, and fear along with the cognitive–emotional states of satisfaction, dissatisfaction, and concentration. The selected emotional states are tested against selected variables gender, department, lecture time, seating positions, and the difficulty of a subject. The proposed system contributes to improve classroom learning.

Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis

AbstractThe importance and pervasiveness of emotions in our lives makes affective computing a tremendously important and vibrant line of work. Systems for automatic emotion recognition (AER) and sentiment analysis can be facilitators of enormous progress (e.g., in improving public health and commerce) but also enablers of great harm (e.g., for suppressing dissidents and manipulating voters). Thus, it is imperative that the affective computing community actively engage with the ethical ramifications of their creations. In this article, I have synthesized and organized information from AI Ethics and Emotion Recognition literature to present fifty ethical considerations relevant to AER. Notably, this ethics sheet fleshes out assumptions hidden in how AER is commonly framed, and in the choices often made regarding the data, method, and evaluation. Special attention is paid to the implications of AER on privacy and social groups. Along the way, key recommendations are made for responsible AER. The objective of the ethics sheet is to facilitate and encourage more thoughtfulness on why to automate, how to automate, and how to judge success well before the building of AER systems. Additionally, the ethics sheet acts as a useful introductory document on emotion recognition (complementing survey articles).

Investigating the Role of Culture on Negative Emotion Expressions in the Wild.

Even though culture has been found to play some role in negative emotion expression, affective computing research primarily takes on a basic emotion approach when analyzing social signals for automatic emotion recognition technologies. Furthermore, automatic negative emotion recognition systems still train data that originates primarily from North America and contains a majority of Caucasian training samples. As such, the current study aims to address this problem by analyzing what the differences are of the underlying social signals by leveraging machine learning models to classify 3 negative emotions, contempt, anger and disgust (CAD) amongst 3 different cultures: North American, Persian, and Filipino. Using a curated data set compiled from YouTube videos, a support vector machine (SVM) was used to predict negative emotions amongst differing cultures. In addition a one-way ANOVA was used to analyse the differences that exist between each culture group in-terms of level of activation of underlying social signal. Our results not only highlighted the significant differences in the associated social signals that were activated for each culture, but also indicated the specific underlying social signals that differ in our cross-cultural data sets. Furthermore, the automatic classification methods showed North American expressions of CAD to be well-recognized, while Filipino and Persian expressions were recognized at near chance levels.

Persian emotion elicitation film set and signal database

Emotions are associated to human central and peripheral nervous system activity and visual stimulus has the potential to invoke emotions. In the field of emotion analysis and recognition, films are widely used due to their similarity to real-life situations and various film sets have been collected in different languages. In this work, we present the PEEFS (Persian Emotion Elicitation Film Set) collection. According to the authors’ knowledge, Persian Emotion Elicitation Film Set (PEEFS) is the first emotion elicitation Persian film set that has been collected and validated by an 88-person population using an online server-based website created in this work. We have watched 84 most-watched Iranian films to trim emotional segments, and the final set consists of 21 film clips, duration from 20 to 100 s, with discrete labeling based on six basic emotions and ratings in two dimensions of arousal and valence. These 21 film clips have the highest intensity at each target emotion. In addition, emotion scores of the film clips are also included. This database can be used to stimulate emotions and to design automatic emotion recognition systems at different levels. In this work, a signal database has been provided by recording 2 channels of forehead EEG (Fp1 and Fp2) and 8 peripheral signals that include ECG lead II, GSR, trapezius and zygomatic EMG (tEMG and zEMG), vertical EOG (vEOG), finger pulse, Respiratory signal and skin temperature, using the PEEFS film set and selected film clips from the previous English and French databases.

Facial Feature Extraction for Emotion Classification Using Fuzzy C-Mean Clustering

Background: Automatic human emotion recognition system is an active area of research due to its wide applications in the field of Human Computer Interaction(HCI) systems, driver fatigue monitoring systems, surveillance systems, human assistance systems, smile detectors etc. Objective: The study presents a fuzzy based approach to extract facial features from input image and builds different classification models to classify the image into two emotion classes i.e. happy and neutral. The system has potential implications in the field of smile detection systems, customer experience analysis and patient monitoring systems. Methods: The proposed system determines the dimensional attributes (l-attribute and w-attribute) of mouth region extracted from the facial image using viola-jones algortithm. The feature set is generated by using a total of 136 images from JAFFE, NimStim and MUG dataset. The differentiating power of the attribures is then evaluated using five different classification models. Results: The accuracy, precision and recall is determined for each classification model. The results show good accuracy of 70% for the grayscale JAFFE and NimStim databases and 95% for the coloured MUG database. Conclusion: The mouth features calculated in the study are based on the geometric coordinates which eliminates the possibility of false distance measurements due to presence of noise or shadows.

Can edges help convolution neural networks in emotion recognition?

Facial emotion recognition has gained importance for its applications in diverse areas. Facial expressions of a subject, when experiencing the same emotion, have wider variations. On the other hand, different subjects experiencing the same emotion may exhibit different facial features. All these make facial emotion recognition challenging. The ability of convolutional neural network (CNN) has been exploited to analyze visual imagery for different applications. It has also been used in developing automatic facial emotion recognition systems. Our objective in this study is to check if an explicit use of edges can help emotion recognition from images using CNN. Edges in an image represent discriminatory information and hence their explicit use is likely to help the training of CNNs and improve emotion recognition. Keeping this in mind we propose a two-tower CNN architecture to classify images into seven basic classes of emotion including the neutral expression. The proposed CNN has an additional tower, called edge-tower, which is simpler in architecture compared to the other tower and it uses edge images as inputs. Our experiments on two benchmark datasets demonstrate that the explicit use of edge information improves the classifier performance.

King Saud University Emotions Corpus: Construction, Analysis, Evaluation, and Comparison

Emotional speech recognition for the Arabic language is insufficiently tackled in the literature compared to other languages. In this paper, we present the work of creating and verifying the King Saud University Emotions (KSUEmotions) corpus, which was released by the Linguistic Data Consortium (LDC) in 2017 as the first public Arabic emotional speech corpus. KSUEmotions contains an emotional speech of twenty-three speakers from Saudi Arabia, Syria, and Yemen, and includes the emotions: neutral, happiness, sadness, surprise, and anger. The corpus content is verified in two different ways: a human perceptual test by nine listeners who rate emotional performance in audio files, and automatic emotion recognition. Two automatic emotion recognition systems are experimented with: Residual Neural Network and Convolutional Neural Network. This work also experiments with emotion recognition for the English language using the Emotional Prosody Speech and Transcripts Corpus (EPST). The current experimental work is conducted in three tracks: (i) monolingual, where independent experiments for Arabic and English are carried out, (ii) multilingual, where the Arabic and English corpora are merged in as mixed corpus, and (iii) cross-lingual, where models are trained using one language and tested using the other. A challenge encountered in this work is that the two corpora do not contain the same emotions. That problem is tackled by mapping the emotions to the arousal-valance space.

Disabled person emotion recognition in EEG signal using deep neural network

Emotion recognition is an important field of research in brain-computer interactions. As technology and the understanding of emotions are advancing, there are growing opportunities for automatic emotion recognition systems. The brain uses the neuromuscular channels to communicate and control its external environment, however many disorders can disrupt these channels. An electroencephalogram (EEG) signals are generated in the human brain, communicate with several neurons and low amplitude signals. In this paper, the hybrid feature extraction (Renyi and differential entropy) was performed on the acquired EEG signal in order to achieve feature subsets. The respective feature values were given as the input for a multi-objective classifier: deep neural network (DNN) for classifying the disabled persons and their emotions. The proposed technique improves the emotion prediction accuracy for different sessions. The emotional recognition classification model includes three states: positive, neutral and negative. In experimental analysis, the proposed approach classifies the disabled persons and their emotions by means of specificity, sensitivity, and accuracy. The experimental outcome shows that the proposed methodology improved accuracy in emotion classification up to 8.01% compared to the existing methods: k-nearest neighbours (KNN) and support vector machine (SVM).

Automatic Emotion Recognition for Cultural Heritage

In this work we present an automatic emotion recognition system for the re-use of multimedia content and storytelling for cultural heritage. A huge amount of heterogeneous multimedia data on cultural heritage is available in online and offline databases that can be used and adapted to produce new content. In the real world, human video editors may want to select the video sequences composing the final video with the intention to induce an emotional reaction in the viewer (e.g. happiness, excitement, sadness). Usually they try to achieve this result following their personal judgement. However, this task of video selection could benefit a lot from the exploitation of an automatic sentiment classification system. Our system can help the editor in choosing the video sequences that best fit the desired emotion to be induced. First-of-all the system splits the video in scenes. Then it classifies them using a multimodal classifier which combines temporal features extracted form LSTM, sentiment-related features obtained through a DNN, audio features and motion-related features. The system learns which features are more important and exploits them to classify the scenes in terms of valence and arousal which are well known to correlate with induced emotions. Finally it provides an online video composer which allows the editor to search, filter and compose the scenes in a new video using sentiment information. To train the classifier we also collected and annotated a small dataset of both users recorded videos and professional ones downloaded from the web.

A comparative study of Autistic Children Emotion recognition based on Spatio-Temporal and Deep analysis of facial expressions features during a Meltdown Crisis

The recognition of human emotion is a significant contribution to many computer vision appli-cations. Despite its importance, this work is the first one towards an automatic Autistic Children emotion recognition system to ensure their security during meltdown crisis. The current solutions to handle a meltdown crisis are based on a preventive approach. Indeed, Meltdown symptoms are determined by abnormal facial expressions related to compound emotions. To provide for this correspondence, we experimentally evaluate, in this paper, hand-crafted Geometric Spatio-Temporal and Deep features of realistic autistic children facial expressions. Towards this end, we compared the Compound Emotion Recognition (CER) performance for different combinations of these features, and we determined the features that best distinguish a Compound Emotion (CE) of autistic children during a meltdown crisis from the normal state. We used “Meltdown crisis”1 dataset to conduct our experiments on realistic Meltdown / Normal scenarios of autistic children. In this evaluation, we show that the gathered features can lead to very encouraging performances through the use of Random Forest classifier (91.27%) with hand-crafted features. Moreover, classifiers trained on deep features from InceptionResnetV2 show higher performance (97.5%) with supervised learning techniques.

Fuzzy-based voiced-unvoiced segmentation for emotion recognition using spectral feature fusions

Despite abundant growth in automatic emotion recognition system (ERS) studies using various techniques in feature extractions and classifiers, scarce sources found to improve the system via pre-processing techniques. This paper proposed a smart pre-processing stage using fuzzy logic inference system (FIS) based on Mamdani engine and simple time-based features i.e. zero-crossing rate (ZCR) and short-time energy (STE) to initially identify a frame as voiced (V) or unvoiced (UV). Mel-frequency cepstral coefficients (MFCC) and linear prediction coefficients (LPC) were tested with K-nearest neighbours (KNN) classifiers to evaluate the proposed FIS V-UV segmentation. We also introduced two feature fusions of MFCC and LPC with formants to obtain better performance. Experimental results of the proposed system surpassed the conventional ERS which yielded a rise in accuracy rate from 3.7% to 9.0%. The fusion of LPC and formants named as SFF LPC-fmnt indicated a promising result between 1.3% and 5.1% higher accuracy rate than its baseline features in classifying between neutral, angry, happy and sad emotions. The best accuracy rates yielded for male and female speakers were 79.1% and 79.9% respectively using SFF MFCC-fmnt fusion technique.