Automatic Emotion Recognition Research Articles

Fidgety Speech Emotion Recognition for Learning Process Modeling

In this paper, the recognition of fidgety speech emotion is studied, and real-world speech emotions are collected to enhance emotion recognition in practical scenarios, especially for cognitive tasks. We first focused on eliciting fidgety emotions and data acquisition for general math learning. Students practice mathematics by performing operations, solving problems, and orally responding to questions, all of which are recorded as audio data. Subsequently, the teacher evaluates the accuracy of these mathematical exercises by scoring, which reflects the cognitive outcomes of the students. Secondly, we propose an end-to-end speech emotion model based on a multi-scale one-dimensional (1-D) residual convolutional neural network. Finally, we conducted an experiment to recognize fidgety speech emotions by testing various classifiers, including SVM, LSTM, 1-D CNN, and the proposed multi-scale 1-D CNN. The experimental results show that the classifier we constructed can identify fidgety emotion well. After conducting a thorough analysis of fidgety emotions and their influence on the learning process, a clear relationship between the two was apparent. The automatic recognition of fidgety emotions is valuable for assisting on-line math teaching.

A robust VGG network combined with Denoising Autoencoder module for human emotion recognition

Human emotion can be divided into multiple categories, which makes it possible to recognize emotions automatically. One critical approach for automated emotion recognition is applying the convolutional neural network to classify emotions on human expression images, but the performance decreases if input distortions occur. This paper introduced a hybrid neural network architecture to make the automated emotion recognition robust towards distorted input images and perform similarly to prediction on clean images. This hybrid neural network combines the Denoise Autoencoder (DAE) network with the Visual Geometry Group (VGG) network. Multiple standalone VGG and Hybrid network experiments were conducted with the control variables method. FER-2013 data set from Kaggle was used as the experimental data set. Distorted input images were generated by adding random noise to clean images. As a result, the research raised a valid hybrid network architecture. The hybrid network improved the emotion classification accuracy on the distorted data set from 16.70% to 57.73%, and the accuracy is similar to the classification result on the clean data set.

Analysis of the evolution of advanced transformer-based language models: experiments on opinion mining

<p>Opinion mining, also known as sentiment analysis, is a subfield of natural language processing (NLP) that focuses on identifying and extracting subjective information in textual material. This can include determining the overall sentiment of a piece of text (e.g., positive or negative), as well as identifying specific emotions or opinions expressed in the text, that involves the use of advanced machine and deep learning techniques. Recently, transformer-based language models make this task of human emotion analysis intuitive, thanks to the attention mechanism and parallel computation. These advantages make such models very powerful on linguistic tasks, unlike recurrent neural networks that spend a lot of time on sequential processing, making them prone to fail when it comes to processing long text. The scope of our paper aims to study the behaviour of the cutting-edge Transformer-based language models on opinion mining and provide a high-level comparison between them to highlight their key particularities. Additionally, our comparative study shows leads and paves the way for production engineers regarding the approach to focus on and is useful for researchers as it provides guidelines for future research subjects.</p>

A Survey on Automated Emotion Recognition Using Different Classification Models and Approaches

Abstract: Automated emotion detection, is a diverse field with a multitude of applications ranging from software engineering to web customization, education, etc. Several methods and approaches have been devised for automatic emotion recognition, which has taken inspiration from human/natural emotion recognition. I have studied and discussed categorical and dimensional models, which can be subdivided into Circumplex, PANA, vector, and Plutchik's models, for defining a myriad of emotions under varied circumstances. I have stratified the approaches used in emotion detection by trifurcating them into lexicon-based, statistical, and hybrid methods. And, I have presented information on different types of classifiers and classes of neural networks that fall under the category of statistical methods, in a systematized way. I have observed that Support Vector Machines provide the most accurate and clear-cut outcome.

Deep Learning Approaches for Classification of Emotion Recognition based on Facial Expressions

Automated emotion recognition (AEE) plays a crucial role in numerous industries that depend on understanding human emotional responses, such as advertising, technology, and human-robot interaction, particularly within the Information Technology (IT) field. However, current systems can often come up short in comprehensively understanding an individual's emotions, as prior research has mainly focused on assessing facial expressions and categorizing them into seven primary emotions, including neutrality. In this study, we present several Deep Convolutional Neural Network (CNN) models designed specifically for the task of facial emotion recognition, utilizing the FER2013 and RAF datasets. The baseline CNN model is established through a trial-and-error method, and its results are compared with more complex deep learning techniques, including ResNet18, VGGNet16, VGGNet19, and Efficient Net-B0 models. Among these models, the VGGNet19 model achieved the best results with a test accuracy of 71.02% on the FER2013 dataset. In comparison, the ResNet18 model outperformed all other models with an 86.02% test accuracy on the RAF-DB dataset. These results underscore the potential for advancing automated emotion recognition through complex deep-learning techniques.

Functional connectivity-enhanced feature-grouped attention network for cross-subject EEG emotion recognition

Electroencephalogram (EEG)-based automatic emotion recognition technologies are gaining significant attention and have become crucial in the field of brain–computer interfaces (BCIs). In particular, deep learning methods have been widely used in emotion recognition in recent years. However, most existing methods generally tend to focus on EEG spatiotemporal information, which ignores the potential relationships between brain activity signals and the differences in functional connectivity under different emotions. Here, we raise a functional connectivity-enhanced feature-grouped attention network (FC-FAN) for cross-subject emotion recognition. The FC-FAN model developed is a dual-input model. One input consists of the differential entropy data derived from the original EEG signals, while the other input comprises the functional connectivity data obtained through the calculation of the phase synchronization index. Then the primary EEG features of the two groups’ input data are extracted through two specific residual blocks. Next, the designed time-series feature grouped attention module (TFGAM) and functional connectivity feature grouped attention model (F2GAM) are utilized to mark interested information or suppress uninterested features for the two groups’ features, respectively. Finally, generated information interacts through a fusion operator. The designed framework could not only sufficiently learn the spatiotemporal features of EEG signals but also clearly analyze nonlinear correlations between electrode signals. Comprehensive tests confirm that the FC-FAN has an excellent effect on subject-independent emotion recognition tasks.

Automated Facial Emotion Recognition Using the Pelican Optimization Algorithm with a Deep Convolutional Neural Network

Facial emotion recognition (FER) stands as a pivotal artificial intelligence (AI)-driven technology that exploits the capabilities of computer-vision techniques for decoding and comprehending emotional expressions displayed on human faces. With the use of machine-learning (ML) models, specifically deep neural networks (DNN), FER empowers the automatic detection and classification of a broad spectrum of emotions, encompassing surprise, happiness, sadness, anger, and more. Challenges in FER include handling variations in lighting, poses, and facial expressions, as well as ensuring that the model generalizes well to various emotions and populations. This study introduces an automated facial emotion recognition using the pelican optimization algorithm with a deep convolutional neural network (AFER-POADCNN) model. The primary objective of the AFER-POADCNN model lies in the automatic recognition and classification of facial emotions. To accomplish this, the AFER-POADCNN model exploits the median-filtering (MF) approach to remove the noise present in it. Furthermore, the capsule-network (CapsNet) approach can be applied to the feature-extraction process, allowing the model to capture intricate facial expressions and nuances. To optimize the CapsNet model’s performance, hyperparameter tuning is undertaken with the aid of the pelican optimization algorithm (POA). This ensures that the model is finely tuned to detect a wide array of emotions and generalizes effectively across diverse populations and scenarios. Finally, the detection and classification of different kinds of facial emotions take place using a bidirectional long short-term memory (BiLSTM) network. The simulation analysis of the AFER-POADCNN system is tested on a benchmark FER dataset. The comparative result analysis showed the better performance of the AFER-POADCNN algorithm over existing models, with a maximum accuracy of 99.05%.

Automatic detection of artifacts and improved classification models for emotional activity detection from multimodal physiological data

This manuscript proposes an automated artifacts detection and multimodal classification system for human emotion analysis from human physiological signals. First, multimodal physiological data, including the Electrodermal Activity (EDA), electrocardiogram (ECG), Blood Volume Pulse (BVP) and respiration rate signals are collected. Second, a Modified Compressed Sensing-based Decomposition (MCSD) is used to extract the informative Skin Conductance Response (SCR) events of the EDA signal. Third, raw features (edge and sharp variations), statistical and wavelet coefficient features of EDA, ECG, BVP, respiration and SCR signals are obtained. Fourth, the extracted raw features, statistical and wavelet coefficient features from all physiological signals are fed into the parallel Deep Convolutional Neural Network (DCNN) to reduce the dimensionality of feature space by removing artifacts. Fifth, the fused artifact-free feature vector is obtained for neutral, stress and pleasure emotion classes. Sixth, an artifact-free feature vector is used to train the Random Forest Deep Neural Network (RFDNN) classifier. Then, a trained RFDNN classifier is applied to classify the test signals into different emotion classes. Thus, leveraging the strengths of both RF and DNN algorithms, more comprehensive feature learning using multimodal psychological data is achieved, resulting in robust and accurate classification of human emotional activities. Finally, an extensive experiment using the Wearable Stress and Affect Detection (WESAD) dataset shows that the proposed system outperforms other existing human emotion classification systems using physiological data.

Analyzing audiovisual data for understanding user's emotion in human−computer interaction environment

PurposeAlthough numerous signal modalities are available for emotion recognition, audio and visual modalities are the most common and predominant forms for human beings to express their emotional states in daily communication. Therefore, how to achieve automatic and accurate audiovisual emotion recognition is significantly important for developing engaging and empathetic human–computer interaction environment. However, two major challenges exist in the field of audiovisual emotion recognition: (1) how to effectively capture representations of each single modality and eliminate redundant features and (2) how to efficiently integrate information from these two modalities to generate discriminative representations.Design/methodology/approachA novel key-frame extraction-based attention fusion network (KE-AFN) is proposed for audiovisual emotion recognition. KE-AFN attempts to integrate key-frame extraction with multimodal interaction and fusion to enhance audiovisual representations and reduce redundant computation, filling the research gaps of existing approaches. Specifically, the local maximum–based content analysis is designed to extract key-frames from videos for the purpose of eliminating data redundancy. Two modules, including “Multi-head Attention-based Intra-modality Interaction Module” and “Multi-head Attention-based Cross-modality Interaction Module”, are proposed to mine and capture intra- and cross-modality interactions for further reducing data redundancy and producing more powerful multimodal representations.FindingsExtensive experiments on two benchmark datasets (i.e. RAVDESS and CMU-MOSEI) demonstrate the effectiveness and rationality of KE-AFN. Specifically, (1) KE-AFN is superior to state-of-the-art baselines for audiovisual emotion recognition. (2) Exploring the supplementary and complementary information of different modalities can provide more emotional clues for better emotion recognition. (3) The proposed key-frame extraction strategy can enhance the performance by more than 2.79 per cent on accuracy. (4) Both exploring intra- and cross-modality interactions and employing attention-based audiovisual fusion can lead to better prediction performance.Originality/valueThe proposed KE-AFN can support the development of engaging and empathetic human–computer interaction environment.

Approach to automatic recognition of emotions in speech transcriptions

Approach to automatic recognition of emotions in speech transcriptions

Objectively Quantifying Pediatric Psychiatric Severity Using Artificial Intelligence, Voice Recognition Technology, and Universal Emotions: Pilot Study for Artificial Intelligence-Enabled Innovation to Address Youth Mental Health Crisis.

Providing Psychotherapy, particularly for youth, is a pressing challenge in the health care system. Traditional methods are resource-intensive, and there is a need for objective benchmarks to guide therapeutic interventions. Automated emotion detection from speech, using artificial intelligence, presents an emerging approach to address these challenges. Speech can carry vital information about emotional states, which can be used to improve mental health care services, especially when the person is suffering. This study aims to develop and evaluate automated methods for detecting the intensity of emotions (anger, fear, sadness, and happiness) in audio recordings of patients' speech. We also demonstrate the viability of deploying the models. Our model was validated in a previous publication by Alemu et al with limited voice samples. This follow-up study used significantly more voice samples to validate the previous model. We used audio recordings of patients, specifically children with high adverse childhood experience (ACE) scores; the average ACE score was 5 or higher, at the highest risk for chronic disease and social or emotional problems; only 1 in 6 have a score of 4 or above. The patients' structured voice sample was collected by reading a fixed script. In total, 4 highly trained therapists classified audio segments based on a scoring process of 4 emotions and their intensity levels for each of the 4 different emotions. We experimented with various preprocessing methods, including denoising, voice-activity detection, and diarization. Additionally, we explored various model architectures, including convolutional neural networks (CNNs) and transformers. We trained emotion-specific transformer-based models and a generalized CNN-based model to predict emotion intensities. The emotion-specific transformer-based model achieved a test-set precision and recall of 86% and 79%, respectively, for binary emotional intensity classification (high or low). In contrast, the CNN-based model, generalized to predict the intensity of 4 different emotions, achieved test-set precision and recall of 83% for each. Automated emotion detection from patients' speech using artificial intelligence models is found to be feasible, leading to a high level of accuracy. The transformer-based model exhibited better performance in emotion-specific detection, while the CNN-based model showed promise in generalized emotion detection. These models can serve as valuable decision-support tools for pediatricians and mental health providers to triage youth to appropriate levels of mental health care services. RR1-10.2196/51912.

MM-EMOR: Multi-Modal Emotion Recognition of Social Media Using Concatenated Deep Learning Networks

Emotion recognition is crucial in artificial intelligence, particularly in the domain of human–computer interaction. The ability to accurately discern and interpret emotions plays a critical role in helping machines to effectively decipher users’ underlying intentions, allowing for a more streamlined interaction process that invariably translates into an elevated user experience. The recent increase in social media usage, as well as the availability of an immense amount of unstructured data, has resulted in a significant demand for the deployment of automated emotion recognition systems. Artificial intelligence (AI) techniques have emerged as a powerful solution to this pressing concern in this context. In particular, the incorporation of multimodal AI-driven approaches for emotion recognition has proven beneficial in capturing the intricate interplay of diverse human expression cues that manifest across multiple modalities. The current study aims to develop an effective multimodal emotion recognition system known as MM-EMOR in order to improve the efficacy of emotion recognition efforts focused on audio and text modalities. The use of Mel spectrogram features, Chromagram features, and the Mobilenet Convolutional Neural Network (CNN) for processing audio data are central to the operation of this system, while an attention-based Roberta model caters to the text modality. The methodology of this study is based on an exhaustive evaluation of this approach across three different datasets. Notably, the empirical findings show that MM-EMOR outperforms competing models across the same datasets. This performance boost is noticeable, with accuracy gains of an impressive 7% on one dataset and a substantial 8% on another. Most significantly, the observed increase in accuracy for the final dataset was an astounding 18%.

Emotion recognition by skeleton-based spatial and temporal analysis

This study introduces an automatic emotion recognition system (AER) focusing on skeletal-based kinematic datasets for enhanced human–computer interaction. Departing from conventional approaches, it achieves real-time emotion recognition in real-life situations. The dataset covers seven emotions and undergoes assessment by eight diverse machine and deep learning algorithms. A thorough investigation is undertaken by varying window sizes and data states, including raw positions and feature-extracted data. The findings imply that incorporating advanced techniques like joint-related feature extraction and robust classifier models yields promising outcomes. Dataset augmentation via varying window sizes enriches insights into real-world scenarios. Evaluations exhibit classification accuracy surpassing 99% for small windows, 94% for medium, and exceeding 88% for larger windows, thereby confirming the robust nature of the approach. Furthermore, we highlight window size's impact on emotion detection and the benefits of combining coordinate axes for efficiency and accuracy. The analysis intricately examines the contributions of features at both the joint and axis levels, assisting in making well-informed selections. The study's contributions include carefully curated datasets, transparent code, and models, all of which ensure the possibility of replication. The paper establishes a benchmark that bridges theory and practicality, solidifying the proposed approach's effectiveness in balancing accuracy and efficiency. By pioneering advanced AER through kinematic data, it sets a new standard for efficacy while driving seamless human–computer interaction through rigorous analysis and strategic design.

EEG-Based Subject-Independent Emotion Recognition Using Gated Recurrent Unit and Minimum Class Confusion

Automatic emotion recognition based on electroencephalogram (EEG) has attracted rapidly increasing interests. Due to large inter-subject variabilities, subject-independent emotion recognition faces great challenges. Recently, domain adaptation methods have been successfully applied in this field due to their ability to align features from different subjects. However, since EEG signals corresponding to some emotions have similar oscillation patterns, they are often confused and aligned to the wrong categories, which limits the generalization ability of the model across subjects. Besides, almost all methods only support offline applications, which require collecting a large number of samples of new subjects. To achieve online recognition, a simpler model is needed. In this paper, a novel Gated Recurrent Unit-Minimum Class Confusion (GRU-MCC) model is proposed. Specifically, a simple feature extractor based on gated recurrent unit (GRU) is firstly applied to model the spatial dependence of multiple electrodes and obtain high-level discriminative features. Then, during training, minimum class confusion (MCC) loss is introduced to reduce the confusion between the correct and ambiguous classes for the target subject and increase the transfer gains. We conduct both offline and online experiments on two public datasets: SEED and MPED. The results indicate that our method can obtain the superior performance.

New Approach for an Affective Computing- Driven Quality of Experience (QoE) Prediction

In human interactions, emotion recognition is crucial. For this reason, the topic of computer-vision approaches for automatic emotion recognition is currently being extensively researched. Processing multi-channel electroencephalogram (EEG) information is one of the most researched methods for automatic emotion recognition. This paper presents a new model for an affective computing-driven Quality of Experience (QoE) prediction. In order to validate the proposed model, a publicly available dataset is used. The dataset contains EEG, ECG, and respiratory data and is focused on a multimedia QoE assessment context. The EEG data are retained on which the differential entropy and the power spectral density are calculated with an observation window of three seconds. These two features were extracted to train several deep-learning models to investigate the possibility of predicting QoE with five different factors. The performance of these models is compared, and the best model is optimized to improve the results. The best results were obtained with an LSTM-based model, presenting an F1-score from 68% to 78%. An analysis of the model and its features shows that the Delta frequency band is the least necessary, that two electrodes have a higher importance, and that two other electrodes have a very low impact on the model's performances.

An Engineering View on Emotions and Speech: From Analysis and Predictive Models to Responsible Human-Centered Applications

The substantial growth of Internet-of-Things technology and the ubiquity of smartphone devices has increased the public and industry focus on speech emotion recognition (SER) technologies. Yet, conceptual, technical, and societal challenges restrict the wide adoption of these technologies in various domains, including, healthcare, and education. These challenges are amplified when automated emotion recognition systems are called to function “in-the-wild” due to the inherent complexity and subjectivity of human emotion, the difficulty of obtaining reliable labels at high temporal resolution, and the diverse contextual and environmental factors that confound the expression of emotion in real life. In addition, societal and ethical challenges hamper the wide acceptance and adoption of these technologies, with the public raising questions about user privacy, fairness, and explainability. This article briefly reviews the history of affective speech processing, provides an overview of current state-of-the-art approaches to SER, and discusses algorithmic approaches to render these technologies accessible to all, maximizing their benefits and leading to responsible human-centered computing applications.

A spatiotemporal symmetrical transformer structure for EEG emotion recognition

In the field of brain-computer interface, automatic recognition of emotions based on electroencephalogram (EEG) signals is of great significance. At present, deep learning can deeply mine the information in the data, especially the convolutional neural network (CNN) and the recurrent neural network (LSTM), which has remarkably improved the accuracy in numerous fields, hence it is applied by researchers to EEG-based emotional identification research. Nonetheless, existing CNN and LSTM-based models still rely on data preprocessing and feature extraction. Furthermore, CNNs have limitations in perceiving global dependencies while LSTMs have problems such as vanishing gradients in the case of long sequences. In this paper, we proposed the Spatiotemporal Symmetric Transformer Model (STS-Transformer), an effective EEG emotion recognition model, to overcome the current problems. STS-Transformer is an end-to-end framework that directly recognizes emotion from raw EEG signals without data preprocessing and feature extraction. This method achieves 89.86% and 86.83% accuracy respectively in the binary classification of valence and arousal in the DEAP dataset; in the DREAMER dataset, the binary classification accuracy of valence and arousal is 85.09% and 82.32%. Therefore, our method exhibits remarkable advantages over other end-to-end models in similar studies.

Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations

Emotion recognition is the ability to precisely infer human emotions from numerous sources and modalities using questionnaires, physical signals, and physiological signals. Recently, emotion recognition has gained attention because of its diverse application areas, like affective computing, healthcare, human–robot interactions, and market research. This paper provides a comprehensive and systematic review of emotion recognition techniques of the current decade. The paper includes emotion recognition using physical and physiological signals. Physical signals involve speech and facial expression, while physiological signals include electroencephalogram, electrocardiogram, galvanic skin response, and eye tracking. The paper provides an introduction to various emotion models, stimuli used for emotion elicitation, and the background of existing automated emotion recognition systems. This paper covers comprehensive searching and scanning of well-known datasets followed by design criteria for review. After a thorough analysis and discussion, we selected 142 journal articles using PRISMA guidelines. The review provides a detailed analysis of existing studies and available datasets of emotion recognition. Our review analysis also presented potential challenges in the existing literature and directions for future research.

A novel variational nonlinear chirp mode decomposition-based critical brain-region investigation for automatic emotion recognition

The field of affective computing that deals with emotion recognition from physiological information, particularly electroencephalography (EEG), is becoming more and more important. Researchers have employed these signals to identify emotions, but the majority of them only distinguish a limited number of emotional states (e.g. happiness, fear, sadness, calmness, anger, etc.). This study outlines the proposed model for classifying nine emotional states from the DREAMER database into arousal, valence, and dominance dimensions using a discrete scale. To create it, we studied variational non-linear chirp mode decomposition (VNCMD) to decompose the EEG signals into five distinct modes. 11 entropy features were extracted from each mode. Then a comparative analysis of six popular machine learning-based classifiers was performed. The VNCMD's third mode with a rotational random forest classifier gave the best performance parameters for this application. This systematic analysis revealed that the best brain region, i.e., the anterior frontal area using only 2 channels, provided the highest accuracy and F1-score for arousal: 96.07% and 97.30%, valence: 94.49% and 95.58%, and dominance: 95.83% and 97.19% categories. Results, using the DREAMER dataset, show that our model can have the highest predictive ability with AUCs of 0.98, 0.98, and 0.97 for the arousal, valence, and dominance categories, respectively. The findings of this study demonstrate that the features, brain regions, and machine learning models commonly employed in emotion classification tasks may be used in more difficult tasks, like the prediction of precise values for arousal, valence, and dominance.

A Deep Learning Approach for Human Facial Expression Recognition using Residual Network – 101

Emotion recognition is a dynamic process that focuses on a person's emotional state, which implies that the emotions associated with each individual's activities are unique. Human emotion analysis and recognition have been popular study areas among computer vision researchers. High dimensionality, execution time, and cost are the main difficulties in human emotion detection. To deal with these issues, the proposed model aims to design a human emotion recognition model using Residual Networks-101 (ResNet-101). A Convolutional Neural Network (CNN) design called ResNet-101 solves the vanishing gradient issue and makes it possible to build networks with thousands of convolutional layers that outperform networks with fewer layers. An image dataset was used for this emotion recognition. Then, this image dataset was subjected to preprocessing to resize the image and eliminate the noise contents present in the images. After preprocessing, the image was given to the classifier to recognize the emotions effectively. Here, ResNet-101 was used for the classification of six classes. The experimental results demonstrate that ResNet-101 models outperform the most recent techniques for emotion recognition. The proposed model was executed in MATLAB software and carried out several performance metrics. The proposed architecture attained better performance in terms of accuracy 92% and error with 0.08 and other performances like 92% of precision, 85% of specificity and 98% of sensitivity so on, and this shows the effectiveness of the proposed model to existing approaches such as LeNet, AlexNet and VGG. In comparison to current techniques, the suggested model provides improved recognition accuracy for low intensity or mild emotional expressions.