Automatic Emotion Recognition Research Articles

Towards Integrating Automatic Emotion Recognition in Education: A Deep Learning Model Based on 5 EEG Channels

In a technologically advanced world, artificial intelligence has impacted all fields of activity. The augmentation of online learning by means of emotion recognition systems raises new challenges in terms of obtaining high-performance systems and in interpreting the results. The paper aims to investigate the usage of automated emotion recognition in learning and to develop a deep learning model based on physiological data to recognize emotions often encountered in classrooms. So, an 1D-CNN model based on physiological data is used to recognize seven emotions: boredom, confusion, frustration, curiosity, excitement, concentration, and anxiety. These emotions are described according to the PAD model and the 5 EEG signals, FP1, AF3, F7, T7, FP2, are taken from the DEAP dataset to train and to test the convolutional neural network model. The high accuracy we obtained (i.e. boredom—99.64%, confusion—99.70%, frustration—99.66%, curiosity—99.80%, excitement—99.91%, concentration—99.70%, anxiety—99.21%) proves that the use of signals obtained via only five channels is sufficient to recognize the presence of emotions. Furthermore, an improved method of analysis based on LIME is proposed and used to obtain reliable explanations for the predictions of our model.

Determining the Optimal Window Duration to Enhance Emotion Recognition Based on Galvanic Skin Response and Photoplethysmography Signals

Automatic emotion recognition using portable sensors is gaining attention due to its potential use in real-life scenarios. Existing studies have not explored Galvanic Skin Response and Photoplethysmography sensors exclusively for emotion recognition using nonlinear features with machine learning (ML) classifiers such as Random Forest, Support Vector Machine, Gradient Boosting Machine, K-Nearest Neighbor, and Decision Tree. In this study, we proposed a genuine window sensitivity analysis on a continuous annotation dataset to determine the window duration and percentage of overlap that optimize the classification performance using ML algorithms and nonlinear features, namely, Lyapunov Exponent, Approximate Entropy, and Poincaré indices. We found an optimum window duration of 3 s with 50% overlap and achieved accuracies of 0.75 and 0.74 for both arousal and valence, respectively. In addition, we proposed a Strong Labeling Scheme that kept only the extreme values of the labels, which raised the accuracy score to 0.94 for arousal. Under certain conditions mentioned, traditional ML models offer a good compromise between performance and low computational cost. Our results suggest that well-known ML algorithms can still contribute to the field of emotion recognition, provided that window duration, overlap percentage, and nonlinear features are carefully selected.

Automatic movie genre classification & emotion recognition via a BiProjection Multimodal Transformer

Analyzing, manipulating, and comprehending data from multiple sources (e.g., websites, software applications, files, or databases) and of diverse modalities (e.g., video, images, audio and text) has become increasingly important in many domains. Despite recent advances in multimodal classification (MC), there are still several challenges to be addressed, such as: the combination of modalities of very diverse nature, the optimal feature engineering for each modality, as well as the semantic alignment between text and images. Accordingly, the main motivation of our research relies in devising a neural architecture that effectively processes and combines text, image, video and audio modalities, so it can offer a noteworthy performance in different MC tasks. In this regard, the Multimodal Transformer (MulT) model is a cutting-edge approach often employed in multimodal supervised tasks, which, although effective, has the problem of having a fixed architecture that limits its performance in specific tasks as well as its contextual understanding, meaning it may struggle to capture fine-grained temporal patterns in audio or effectively model spatial relationships in images. To address these issues, our research modifies and extends the MulT model in several aspects. Firstly, we focus on leveraging the Gated Multimodal Unit (GMU) module within the architecture to efficiently and dynamically weigh modalities at the instance level and to visualize the use of modalities. Secondly, to overcome the problem of vanishing and exploding gradients we focus on strategically placing residual connections in the architecture. The proposed architecture is evaluated in two different and complex classification tasks, on the one hand, the movie genre categorization (MGC) and, on the other hand, the multimodal emotion recognition (MER). The results obtained are encouraging as they indicate that the proposed architecture is competitive against state-of-the-art (SOTA) models in MGC, outperforming them by up to 2% on the Moviescope dataset, and by 1% on the MM-IMDB datasets. Furthermore, in the MER task the unaligned version of the datasets was employed, which is considerably more difficult; we improve accuracy SOTA results by up to 1% on the IEMOCAP dataset, and attained a competitive outcome on the CMU-MOSEI11Dai et al., (2021) collection, outperforming SOTA results in several emotions.

Facial Expression Analysis Using Multi-label Classification

Emotions form the reflection of an individual's mental state. Automatic expression analysis and its recognition based on their facial landmarks are of critical use, particularly to understand the emotional states of physically disabled individuals like autistic children, deaf, dumb, and bedridden individuals. This approach helps in interpreting human emotions, which is crucial for designing computational models and interfaces that cater to their needs. While traditional research has focused on six basic emotion categories (happiness, surprise, sadness, anger, fear, disgust), your work expands this by exploring compound emotion categories. Compound emotions combine these basic categories to create nuanced emotional states. The research utilizes the CFEE_Database_230 dataset of facial expressions captured for analysis and training purposes. The proposed methodology has the following three steps: Analyze the dataset and extract the region of interest (ROI), Extract various statistical and potentially discriminating features and apply a multi-label classification approach to categorize sets of emotions. This involves comparing the feature values across different emotion classes and assigning appropriate labels, particularly focusing on compound emotion categories. Also, the research employs different classifiers and metrics to evaluate the effectiveness of the model. After applying the classification methods, the results are analyzed using various metrics to assess the accuracy and effectiveness of emotion recognition based on facial landmarks. Based on the findings, the Binary Relevance Method yielded the best performance with Mean Average Precision and hamming Loss of 0.7227±0.0344 and 0.1906±0.0227, respectively. Overall, the work contributes to advancing automatic emotion recognition by considering a broader range of emotional categories beyond the traditional basics. This is particularly beneficial for populations such as the physically disabled and autistic children, where traditional communication methods might be limited or challenging.

TFCNN-BiGRU with self-attention mechanism for automatic human emotion recognition using multi-channel EEG data

TFCNN-BiGRU with self-attention mechanism for automatic human emotion recognition using multi-channel EEG data

Bridging computer and education sciences: A systematic review of automated emotion recognition in online learning environments

Emotions play an important role in the learning process. With intelligent technology support, identification and intervention of learners’ cognition have made great achievement, but the care of emotion has been in the absence for a long time. In recent years, the use of affective computing technology to solve affective loss in online education has become a key research topic. To date, a growing number of studies have investigated automated emotion recognition (AER) in online environments. However, AER has been mainly studied from the perspective of computer science focusing on technical characteristics of developing AI technology while its pedagogical value and educational application has been overlooked. Therefore, this systematic literature review aimed to bring together educational and technical aspects of AER. Following PRISMA methodology, a comprehensive search of AER research from 2010 to 2024 in three databases (Web of Science, Science Direct and IEEE Xplore) identified 117 studies that met inclusion criteria. The articles were coded for report characteristics, educational characteristics (tech platform, pedagogy, assessment, content), technical characteristics (emotion model, emotion category, emotion measurement channel, database, algorithm model) and outcome characteristics (technical result, educational application). We found that the primary purpose of these studies was to develop and evaluate systems for AER, rather than implementing these systems in real online learning environments. Furthermore, our findings indicated a lack of integration between computer science and educational science in the realm of AER. Despite the fact that most algorithm models demonstrated high accuracy in AER, the interpretability of the results was significantly constrained by the quality of the databases used, along with the scarcity of studies focusing on the effective and real-time application of AER results. These findings provide essential guidance for shaping future research and development pathways in this field.

Advanced techniques for automated emotion recognition in dogs from video data through deep learning

Inter-species emotional relationships, particularly the symbiotic interaction between humans and dogs, are complex and intriguing. Humans and dogs share fundamental mammalian neural mechanisms including mirror neurons, crucial to empathy and social behavior. Mirror neurons are activated during the execution and observation of actions, indicating inherent connections in social dynamics across species despite variations in emotional expression. This study explores the feasibility of using deep-learning Artificial Intelligence systems to accurately recognize canine emotions in general environments, to assist individuals without specialized knowledge or skills in discerning dog behavior, particularly related to aggression or friendliness. Starting with identifying key challenges in classifying pleasant and unpleasant emotions in dogs, we tested advanced deep-learning techniques and aggregated results to distinguish potentially dangerous human--dog interactions. Knowledge transfer is used to fine-tune different networks, and results are compared on original and transformed sets of frames from the Dog Clips dataset to investigate whether DogFACS action codes detailing relevant dog movements can aid the emotion recognition task. Elaborating on challenges and biases, we emphasize the need for bias mitigation to optimize performance, including different image preprocessing strategies for noise mitigation in dog recognition (i.e., face bounding boxes, segmentation of the face or body, isolating the dog on a white background, blurring the original background). Systematic experimental results demonstrate the system’s capability to accurately detect emotions and effectively identify dangerous situations or signs of discomfort in the presence of humans.

Development of a method for recognizing emotions from a speech signal

The purpose of research is automatic recognition of the speaker's emotions, based on the processing of sound recordings intended for use in alarm systems when working with operators of locomotive crews and dispatch services.Methods. Human emotion recognition has been a rapidly developing area of research in recent years. Features of the vocal tract, such as sound power, formant frequencies, are used to detect certain emotions with good accuracy. A method was used to determine the signal energy by highlighting the dominant frequency. The work has developed a program code, on the basis of which an analysis of four emotions is given - anger, joy, fear and calm. The most important and difficult step is to determine the features most suitable for distinguishing emotions and the availability of databases. Collecting databases is a complex task requiring the manifestation of sincerity of emotions. Often, the collection of a database takes place in an artificial environment and the speech may sound staged; to eliminate such problems, it is necessary to use call center recordings.Results. Recordings of basic emotional states, such as anger, joy, sadness, fear and surprise, which are the most common case of the study, were obtained and processed. The developed software code allows us to get closer to automatically determining emotions from a speech signal. To analyze speech recordings in samples, indicators of signal energy and identification of the dominant frequency were used.Conclusion. The implemented method of monitoring the emotional state of a human operator using a speech signal is widely used in the prevention and improvement of indicators of the psychophysiological professional suitability of locomotive crew workers and the preservation of their professional health. Distinct differences are observed in the characteristics of all types of emotions.

Analysis of EEG Fluctuation Patterns Using Nonlinear Phase-Based Functional Connectivity Measures for Emotion Recognition

Automated emotion recognition is crucial in identifying and monitoring psychological disorders. Although several electroencephalography (EEG)-based methods have been explored for emotion recognition, capturing the subtle oscillations within EEG signals associated with distinct emotional states remains a persistent challenge. Nonlinear phase-based functional connectivity (FC) can capture the intricate time-varying patterns of brain activity during the processing of emotions. In this work, an attempt has been made to characterize the EEG-based emotional states using nonlinear phase-based FC techniques. For this, the EEG signals are obtained from the publicly available DEAP database and decomposed into four frequency bands: Theta (4–7[Formula: see text]Hz), alpha (8–12[Formula: see text]Hz), beta (13–30[Formula: see text]Hz) and gamma (30–45[Formula: see text]Hz). Three nonlinear phase-based FC measures, namely phase lag index (PLI), phase locking value (PLV) and Rho index, are extracted from individual frequency bands. Two types of features, namely network features and FC indices, are fed to three classifiers, namely random forest (RF), extreme gradient boosting (XGB) and K-Nearest Neighbors (KNN). The results reveal that the proposed approach can capture EEG dynamics to characterize emotional states. The gamma band-based Rho index demonstrated prominence in discriminating arousal and valence emotional states. The utilization of the Rho index-based FC feature effectively reveals interactions among cortical brain regions in response to audio-visual stimuli. Thus, the proposed approach could be extended to classifying various emotional states in clinical settings.

Emotion classification using electrocardiogram and machine learning: A study on the effect of windowing techniques

Automated emotion recognition using physiological signals has gained significant attention in recent years due to its potential applications in human–computer interaction, healthcare, and psychology. Electrocardiogram (ECG) signals are widely used due to their non-invasiveness, high temporal resolution, and direct relationship with the autonomic nervous system. In this study, we propose a novel approach for ECG-based emotion recognition using time-series to image encoding techniques and texture-based features combined with machine learning algorithms and deep learning architectures. The ECG data used in this study were obtained from the Continuously Annotated Signals of Emotion (CASE) and Wearable Stress and Affect Detection (WESAD) datasets. We categorized emotional states based on valence and arousal annotations into four classes: High-Valence High-Arousal, High-Valence Low-Arousal, Low-Valence High-Arousal, and Low-Valence Low-Arousal. The ECGs were segmented into 5 and 7-window segments and transformed into 2D representations using Gramian Angular Summation Field, Markov Transition Field, Recurrence Plot (RP), and the triple-channel fusion of all these images. Total of 85 textural features based on the Gray-Level Co-occurrence Matrix (GLCM), Gray-Level Run Length Matrix, Zernike’s Moments, Hu’s Moments, Fractal Dimension Texture Analysis (FDTA), and First-Order Statistics were extracted. Classifiers, namely Random Forest, Support Vector Machine (SVM), eXtreme Gradient Boosting (XGB), 1D Convolutional Neural Network (CNN), and Multi-head Attention Network were considered to classify the emotional states. The performance of the classifiers varied depending on the time-series to image encoding technique, segmentation approaches, and the classifier employed. We achieved the highest Weighted F-measure (F-m) of 94.91% (RP + XGB) and 86.78% (RP + SVM) using the 7-window and 5-window approaches, respectively. Our proposed 1D CNN architecture achieved the highest classification metrics (F-m = 92.52%, Balanced accuracy = 92.0%, Recall = 91.96%, and Precision = 93.16%) with RP images in a 7-window approach. GLCM and FDTA features made significant contributions to the classification of emotional states. Overall, our results suggest that the proposed method holds promise for developing more accurate and efficient emotion recognition systems.

Sentiment Analysis-Based Automatic Stress and Emotion Recognition using Weighted Fused Fusion-Based Cascaded DTCN with Attention Mechanism from EEG Signal

When loaded with difficulties in fulfilling daily requirements, a lot of people in today’s world experience an emotional pressure known as stress. Stress that lasts for a short duration of time has more advantages as they are good for mental health. But, the persistence of stress for a long duration of time may lead to serious health impacts in individuals, such as high blood pressure, cardiovascular disease, stroke and so on. Long-term stress, if unidentified and not treated, may also result in personality disorder, depression and anxiety. The initial detection of stress has become more important to prevent the health issues that arise due to stress. Detection of stress based on brain signals for analysing the emotion in humans leads to accurate detection outcomes. Using EEG-based detection systems and disease, disability and disorders can be identified from the brain by utilising the brain waves. Sentiment Analysis (SA) is helpful in identifying the emotions and mental stress in the human brain. So, a system to accurately and precisely detect depression in human based on their emotion through the utilisation of SA is of high necessity. The development of a reliable and precise Emotion and Stress Recognition (ESR) system in order to detect depression in real-time using deep learning techniques with the aid of Electroencephalography (EEG) signal-based SA is carried out in this paper. The essentials needed for performing stress and emotion detection are gathered initially from benchmark databases. Next, the pre-processing procedures, like the removal of artifacts from the gathered EEG signal, are carried out on the implemented model. The extraction of the spectral attributes is carried out from the pre- processed. The extracted spectral features are considered the first set of features. Then, with the aid of a Conditional Variational Autoencoder (CVA), the deep features are extracted from the pre-processed signals forming a second set of features. The weights are optimised using the Adaptive Egret Swarm Optimisation Algorithm (AESOA) so that the weighted fused features are obtained from these two sets of extracted features. Then, a Cascaded Deep Temporal Convolution Network with Attention Mechanism (CDTCN-AM) is used to recognise stress and emotion. The validation of the results from the developed stress and emotion recognition approach is carried out against traditional models in order to showcase the effectiveness of the suggested approach.

Smart 6G Sensor Network Based Human Emotion Analysis by Machine Learning Architectures

Smart 6G Sensor Network Based Human Emotion Analysis by Machine Learning Architectures

Crowds and Smiles: Visual Opportunity Structures and the Communication of European Political Leaders During the COVID-19 Pandemic

ABSTRACT The growing interest in political leaders’ visual communication often emphasizes specific visual features without focusing on the driving factors behind these strategies. Our study introduces the Visual Opportunity Structure (VOS) theory, aiming to explain the use of specific visual elements based on their suitability within the socio-political context. We examined the COVID-19 pandemic, analyzing a large dataset (N = 73,379) of Instagram posts by 28 European national party leaders coded through automatic facial and emotional recognition algorithms. The findings reveal a negative link between the use of inappropriate visual features during pandemic waves, like depicting happiness and groups of people, and the severity of the pandemic’s impact. Political leaders significantly reduce these inappropriate visuals during severe waves, reintroducing them in calmer periods. This trend is particularly pronounced among government party leaders. Our research not only unveils a pattern in the visual communication tactics used by political figures during the pandemic but also provides deeper insights into how visual strategies align with the broader context. By shedding light on these nuances, the study contributes to a more comprehensive understanding of visual political communication online.

Emotion and Gesture Recognition

Human Gestures and emotions play an important role in interpersonal relationships. The automatic recognition of emotions and gestures has been an active research topic from early eras. Emotions are reflected from facial expression, speech and gesture of the body. Hence understanding of emotion and gesture has high importance in interaction between humans(human-human) as well as between human and machine communication.In this system captured images are compared with the trained dataset available in the database and then display emotional state and gesture. For this system, fer13 (Facial Expressions Recognition 2013 dataset) and VGG16 (Visual Geometry Group) datasets are used to compare with captured images. In this system emotion and gesture recognition, both will run simultaneously. EDA(Exploratory Data Analysis) is used to analyse and validate the train data for both emotion as well as gesture dataset.

Emotion recognition with reduced channels using CWT based EEG feature representation and a CNN classifier

Objective. Although emotion recognition has been studied for decades, a more accurate classification method that requires less computing is still needed. At present, in many studies, EEG features are extracted from all channels to recognize emotional states, however, there is a lack of an efficient feature domain that improves classification performance and reduces the number of EEG channels. Approach. In this study, a continuous wavelet transform (CWT)-based feature representation of multi-channel EEG data is proposed for automatic emotion recognition. In the proposed feature, the time-frequency domain information is preserved by using CWT coefficients. For a particular EEG channel, each CWT coefficient is mapped into a strength-to-entropy component ratio to obtain a 2D representation. Finally, a 2D feature matrix, namely CEF2D, is created by concatenating these representations from different channels and fed into a deep convolutional neural network architecture. Based on the CWT domain energy-to-entropy ratio, effective channel and CWT scale selection schemes are also proposed to reduce computational complexity. Main results. Compared with previous studies, the results of this study show that valence and arousal classification accuracy has improved in both 3-class and 2-class cases. For the 2-class problem, the average accuracies obtained for valence and arousal dimensions are 98.83% and 98.95%, respectively, and for the 3-class, the accuracies are 98.25% and 98.68%, respectively. Significance. Our findings show that the entropy-based feature of EEG data in the CWT domain is effective for emotion recognition. Utilizing the proposed feature domain, an effective channel selection method can reduce computational complexity.

Machine Learning Based Speech Emotion Recognition System

In the last decade, there has been significant research into Automatic Speech Emotion Recognition (SER). The primary goal of SER is to improve human-machine interfaces. It can also monitor someone's psychological state for lie detection applications. Recently, speech emotion recognition has found uses in medicine and forensics. This paper recognizes 7 emotions using pitch and prosody features. The majority of speech features used here are in the time domain. A Support Vector Machine (SVM) classifier categorizes the emotions. The Berlin emotional database was used for this task. A good recognition rate of 81% was achieved. The reference paper for this work recognized 4 emotions and obtained a 94.2% recognition rate. However, the reference paper used a more complex hybrid classifier, while this work focuses on recognizing more emotions with a simpler model. . KeyWords: Blindness, Smart Stick, BlinDar, GPS, ESP8266, Internet of Things, RF Tx/Rx, MQ2.

Exploring the Impact of Convolutional Neural Networks on Facial Emotion Detection and Recognition

Emotional analytics is a fascinating blend of psychology and technology, with one of the primary methods for recognizing emotions involving facial expression analysis. Facial emotion detection has advanced significantly, utilizing deep learning algorithms to identify common emotions. In recent years, substantial progress has been made in automatic facial emotion recognition (FER). This technology has been applied across various industries to enhance interactions between humans and machines, particularly in human-centered computing and the emerging field of emotional artificial intelligence (EAI). Researchers focus on improving systems’ capabilities to recognize and interpret human facial expressions and behaviors in diverse contexts. The impact of convolutional neural networks (CNNs) on this field has been profound, as these networks have undergone significant development, leading to diverse architectures designed to address increasingly complex challenges. This article explores the latest advancements in automated emotion recognition using computational intelligence, emphasizing how contemporary deep learning models contribute to the field. It provides a review of recent developments in CNN architectures for FER over the past decade, demonstrating how deep learning-based methods and specialized databases collaborate to achieve highly accurate outcomes.

HERF: A Machine Learning Framework for Automatic Emotion Recognition from Audio

HERF: A Machine Learning Framework for Automatic Emotion Recognition from Audio

A multi-featured expression recognition model incorporating attention mechanism and object detection structure for psychological problem diagnosis

Expression is the main method for judging the emotional state and psychological condition of the human body, and the prediction of changes in facial expressions can effectively determine the mental health of a person, thus avoiding serious psychological or psychiatric disorders due to early negligence. From a computer vision perspective, most researchers have focused on studying facial expression analysis, and in some cases, body posture is also considered. However their performance is more limited under unconstrained natural conditions, which requires more information to be used in human emotion analysis. In this paper, we design an Adaptive Multi-End Fusion Attention Mechanism suitable for extracting human body information based on the deep learning framework, depending on human expressions, postures and the environment they are in and add it to an object detection model to obtain the information we need from different regions of the human body and face and features of different sizes and use fusion networks for feature fusion and classification, and from different test methods to confirm that this fusion approach for expression recognition and prediction is feasible. This model achieves an average accuracy of 34.51 % in the Emotic contextual expression recognition dataset.

Evaluation of Eye-Blinking Dynamics in Human Emotion Recognition Using Weighted Visibility Graph

Purpose: Designing an automated emotion recognition system using biosignals has become a hot and challenging issue in many fields, including human-computer interferences, robotics, and affective computing. Several algorithms have been proposed to characterize the internal and external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are habitually analyzed in a multi-modality system using classic statistical measures, and the evaluation of its dynamics has been neglected so far.
 Materials and Methods: This experiment intended to provide an innovative single-modality scheme for emotion classification using eye-blinking data. The dynamics of eye-blinking data have been characterized by weighted visibility graph-based indices. The extracted measures were then fed to the different classifiers, including support vector machine, decision tree, k-Nearest neighbor, Adaptive Boosting, and random subset to complete the process of classifying sad, happy, neutral, and fearful affective states. The scheme has been evaluated utilizing the available signals in the SEED-IV database.
 Results: The proposed framework provided significant performance in terms of recognition rates. The highest average recognition rates of > 90% were achieved using the decision tree.
 Conclusion: In brief, our results showed that eye-blinking data has the potential for emotion recognition. The present system can be extended for designing future affect recognition systems.