Human Emotion Recognition Research Articles

Feature fusion for human compound emotion recognition: a fusion of facial expression texture and action unit data

Feature fusion for human compound emotion recognition: a fusion of facial expression texture and action unit data

Generative technology for human emotion recognition: A scoping review

Affective computing stands at the forefront of artificial intelligence (AI), seeking to imbue machines with the ability to comprehend and respond to human emotions. Central to this field is emotion recognition, which endeavors to identify and interpret human emotional states from different modalities, such as speech, facial images, text, and physiological signals. In recent years, important progress has been made in generative models, including Autoencoder, Generative Adversarial Network, Diffusion Model, and Large Language Model. These models, with their powerful data generation capabilities, emerge as pivotal tools in advancing emotion recognition. However, up to now, there remains a paucity of systematic efforts that review generative technology for emotion recognition. This survey aims to bridge the gaps in the existing literature by conducting a comprehensive analysis of over 330 research papers until June 2024. Specifically, this survey will firstly introduce the mathematical principles of different generative models and the commonly used datasets. Subsequently, through a taxonomy, it will provide an in-depth analysis of how generative techniques address emotion recognition based on different modalities in several aspects, including data augmentation, feature extraction, semi-supervised learning, cross-domain, etc. Finally, the review will outline future research directions, emphasizing the potential of generative models to advance the field of emotion recognition and enhance the emotional intelligence of AI systems.

Multimodal fusion: A study on speech-text emotion recognition with the integration of deep learning

Recognition of various human emotions holds significant value in numerous real-world scenarios. This paper focuses on the multimodal fusion of speech and text for emotion recognition. A 39-dimensional Mel-frequency cepstral coefficient (MFCC) was used as a feature for speech emotion. A 300-dimensional word vector obtained through the Glove algorithm was used as the feature for text emotion. The bidirectional gate recurrent unit (BiGRU) method in deep learning was added for extracting deep features. Subsequently, it was combined with the multi-head self-attention (MHA) mechanism and the improved sparrow search algorithm (ISSA) to obtain the ISSA-BiGRU-MHA method for emotion recognition. It was validated on the IEMOCAP and MELD datasets. It was found that MFCC and Glove word vectors exhibited superior recognition effects as features. Comparisons with the support vector machine and convolutional neural network methods revealed that the ISSA-BiGRU-MHA method demonstrated the highest weighted accuracy and unweighted accuracy. Multimodal fusion achieved weighted accuracies of 76.52 %, 71.84 %, 66.72 %, and 62.12 % on the IEMOCAP, MELD, MOSI, and MOSEI datasets, suggesting better performance than unimodal fusion. These results affirm the reliability of the multimodal fusion recognition method, showing its practical applicability.

ER-GET: Emotion Recognition Based on Global ECG Trajectory.

In recent years, the recognition of human emotions based on electrocardiogram (ECG) signals has been considered a novel area of study among researchers. Despite the challenge of extracting latent emotion information from ECG signals, existing methods are able to recognize emotions by calculating the heart rate variability (HRV) features. However, such local features have drawbacks, as they do not provide a comprehensive description of ECG signals, leading to suboptimal recognition performance. For the first time, we propose a new strategy to extract hidden emotional information from the global ECG trajectory for emotion recognition. Specifically, a period of ECG signals is decomposed into sub-signals of different frequency bands through ensemble empirical mode decomposition (EEMD), and a series of multi-sequence trajectory graphs is constructed by orthogonally combining these sub-signals to extract latent emotional information. Additionally, to better utilize these graph features, a network has been designed that includes self-supervised graph representation learning and ensemble learning for classification. This approach surpasses recent notable works, achieving outstanding results, with an accuracy of 95.08% in arousal and 95.90% in valence detection. Additionally, this global feature is compared and discussed in relation to HRV features, with the intention of providing inspiration for subsequent research.

ARNT2 controls prefrontal somatostatin interneurons mediating affective empathy

Empathy, crucial for social interaction, is impaired across various neuropsychiatric conditions. However, the genetic and neural underpinnings of empathy variability remain elusive. By combining forward genetic mapping with transcriptome analysis, we discover that aryl hydrocarbon receptor nuclear translocator 2 (ARNT2) is a key driver modulating observational fear, a basic form of affective empathy. Disrupted ARNT2 expression in the anterior cingulate cortex (ACC) reduces affect sharing in mice. Specifically, selective ARNT2 ablation in somatostatin (SST)-expressing interneurons leads to decreased pyramidal cell excitability, increased spontaneous firing, aberrant Ca2+ dynamics, and disrupted theta oscillations in the ACC, resulting in reduced vicarious freezing. We further demonstrate that ARNT2-expressing SST interneurons govern affective state discrimination, uncovering a potential mechanism by which ARNT2 polymorphisms associate with emotion recognition in humans. Our findings advance our understanding of the molecular mechanism controlling empathic capacity and highlight the neural substrates underlying social affective dysfunctions in psychiatric disorders.

An Interactive Web Interface for Facial Expression Recognition Using Deep Learning Framework: A Review

Facial expression is the instant reaction which shows on the face in a very short time. Visual system and brain easily understands the emotions of the person. Many computer vision researchers struggle to automate human emotion recognition. Facial expression recognition plays an important role in the area of human computer interaction, camera surveillance, security, identity verification in modern electronic devices. Working of human brain with neurons is represented using deep learning paradigm. This learning usually consists of a neural network model where neurons act as input and each of them are connected to move as outputs. In deep learning number of level are available which provides the interpretation of the data inserted. Proposed system use a Python-based framework that provides access to deep learning libraries such as TensorFlow and Keras. We are going to develop developed an interactive web interface to enable ease of use, scalability and adaptability of the system to different use-case scenarios. Here we have studied more than 30 papers of standard publications which are published after 2020 so as to have a recent ongoing research in this topic.

Multimodal In‐Sensor Computing Implemented by Easily‐Fabricated Oxide‐Heterojunction Optoelectronic Synapses

AbstractThe advent of big data and the Internet of Things has created urgent demands for in‐sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple‐structured optoelectronic synaptic device of In2O3·SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in‐sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in‐sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single‐modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real‐time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low‐cost and easy to mass‐produce, but also paves the way for next‐generation in‐sensor computing that can efficiently process multimodal signals.

Sensing Human Emotion using Emerging Machine Learning Techniques

Human emotion recognition using machine learning is a new field that has the potential to improve user experience, lower crime, and target advertising. The ability of today's emotion detection systems to identify human emotions is essential. Applications ranging from security cameras to emotion detection are readily accessible. Machine learning-based emotion detection recognises and deciphers human emotions from text and visual data. In this study, we use convolutional neural networks and natural language processing approaches to create and assess models for emotion detection. Instead of speaking clearly, these human face expressions visually communicate a lot of information. Recognising facial expressions is important for human-machine interaction. Applications for automatic facial expression recognition systems are numerous and include, but are not limited to, comprehending human conduct, identifying mental health issues, and creating artificial human emotions. It is still difficult for computers to recognise facial expressions with a high recognition rate. Geometry and appearance-based methods are two widely used approaches for automatic FER systems in the literature. Pre-processing, face detection, feature extraction, and expression classification are the four steps that typically make up facial expression recognition. The goal of this research is to recognise the seven main human emotions anger, disgust, fear, happiness, sadness, surprise, and neutrality using a variety of deep learning techniques (convolutional neural networks).

TFCNN-BiGRU with self-attention mechanism for automatic human emotion recognition using multi-channel EEG data

TFCNN-BiGRU with self-attention mechanism for automatic human emotion recognition using multi-channel EEG data

Development of a method for recognizing emotions from a speech signal

The purpose of research is automatic recognition of the speaker's emotions, based on the processing of sound recordings intended for use in alarm systems when working with operators of locomotive crews and dispatch services.Methods. Human emotion recognition has been a rapidly developing area of research in recent years. Features of the vocal tract, such as sound power, formant frequencies, are used to detect certain emotions with good accuracy. A method was used to determine the signal energy by highlighting the dominant frequency. The work has developed a program code, on the basis of which an analysis of four emotions is given - anger, joy, fear and calm. The most important and difficult step is to determine the features most suitable for distinguishing emotions and the availability of databases. Collecting databases is a complex task requiring the manifestation of sincerity of emotions. Often, the collection of a database takes place in an artificial environment and the speech may sound staged; to eliminate such problems, it is necessary to use call center recordings.Results. Recordings of basic emotional states, such as anger, joy, sadness, fear and surprise, which are the most common case of the study, were obtained and processed. The developed software code allows us to get closer to automatically determining emotions from a speech signal. To analyze speech recordings in samples, indicators of signal energy and identification of the dominant frequency were used.Conclusion. The implemented method of monitoring the emotional state of a human operator using a speech signal is widely used in the prevention and improvement of indicators of the psychophysiological professional suitability of locomotive crew workers and the preservation of their professional health. Distinct differences are observed in the characteristics of all types of emotions.

A Novel Hybrid Feature Extraction Methods for EEG Based Emotions Classification Using Ensemble Approaches in HIED Dataset

Introduction: Emotions play a significant role in human relationships in everyday challenges. Human-machine interactions and the complexities of its research include humanoid robots which rely significantly on emotional queries. The present study recommends a novel methodology for human emotion identification using electroencephalogram (EEG) signals. Machine learning (ML) techniques based on electroencephalogram have demonstrated impressive results in distinguishing emotions. Method: The present study investigates five ensemble learning-based machine learning (EML) and five conventional machine learning (CML) algorithms for achieving the recognition of human emotions from EEG signals. The main objective of the current research study is to implement the optimal approach for classifying emotions and subsequently will achieve it by using physiological characteristics to categorise the seven main emotional states (stress, happy, sadness, anger, fear, disgust, and surprise). This research follows the study investigation of ensemble learning-based machine learning (EML) and conventional machine learning (CML) algorithms for achieving the recognition of human emotions from EEG signals. Initially, the EEG data in this study are categorized into theta, alpha, beta, and gamma bands using a DWT. Result: With this idea, the research elaborates on further decomposition based on bandseparated EEG signals known as intrinsic mode functions (IMFs) along with empirical mode decomposition is subsequently implemented. Then, 13 statistical characteristics are extracted from the IMFs using five multiclass EML algorithms: adaptive boosting, rotation random forest, bagging, random forest, and extreme gradient boost. The following procedure is carried out to construct an ML-based system. In the final phase, 10-fold cross-validation is used to assess the efficacy of these five EML algorithms. Accuracy, F1-score, and area-under-the-curve (AUC) are performance assessment metrics that are employed to compare the final results of these algorithms with the five CML techniques. Conclusion: The proposed method for EEG-based emotion analysis is significantly more effective and achieves greater outcomes than all of the techniques that were compared, based on the data from the experiments.

GC-STCL: A Granger Causality-Based Spatial-Temporal Contrastive Learning Framework for EEG Emotion Recognition.

EEG signals capture information through multi-channel electrodes and hold promising prospects for human emotion recognition. However, the presence of high levels of noise and the diverse nature of EEG signals pose significant challenges, leading to potential overfitting issues that further complicate the extraction of meaningful information. To address this issue, we propose a Granger causal-based spatial-temporal contrastive learning framework, which significantly enhances the ability to capture EEG signal information by modeling rich spatial-temporal relationships. Specifically, in the spatial dimension, we employ a sampling strategy to select positive sample pairs from individuals watching the same video. Subsequently, a Granger causality test is utilized to enhance graph data and construct potential causality for each channel. Finally, a residual graph convolutional neural network is employed to extract features from EEG signals and compute spatial contrast loss. In the temporal dimension, we first apply a frequency domain noise reduction module for data enhancement on each time series. Then, we introduce the Granger-Former model to capture time domain representation and calculate the time contrast loss. We conduct extensive experiments on two publicly available sentiment recognition datasets (DEAP and SEED), achieving 1.65% improvement of the DEAP dataset and 1.55% improvement of the SEED dataset compared to state-of-the-art unsupervised models. Our method outperforms benchmark methods in terms of prediction accuracy as well as interpretability.

Predicting game-induced emotions using EEG, data mining and machine learning

BackgroundEmotion is a complex phenomenon that greatly affects human behavior and thinking in daily life. Electroencephalography (EEG), one of the human physiological signals, has been emphasized by most researchers in emotion recognition as its specific properties are closely associated with human emotion. However, the number of human emotion recognition studies using computer games as stimuli is still insufficient as there were no relevant publicly available datasets provided in the past decades. Most of the recent studies using the Gameemo public dataset have not clarified the relationship between the EEG signal’s changes and the emotion elicited using computer games. Thus, this paper is proposed to introduce the use of data mining techniques in investigating the relationships between the frequency changes of EEG signals and the human emotion elicited when playing different kinds of computer games. The data acquisition stage, data pre-processing, data annotation and feature extraction stage were designed and conducted in this paper to obtain and extract the EEG features from the Gameemo dataset. The cross-subject and subject-based experiments were conducted to evaluate the classifiers’ performance. The top 10 association rules generated by the RCAR classifier will be examined to determine the possible relationship between the EEG signal's frequency changes and game-induced emotions.ResultsThe RCAR classifier constructed for cross-subject experiment achieved highest accuracy, precision, recall and F1-score evaluated with over 90% in classifying the HAPV, HANV and LANV game-induced emotions. The 20 experiment cases’ results from subject-based experiments supported that the SVM classifier could accurately classify the 4 emotion states with a kappa value over 0.62, demonstrating the SVM-based algorithm’s capabilities in precisely determining the emotion label for each participant’s EEG features’ instance.ConclusionThe findings in this study fill the existing gap of game-induced emotion recognition field by providing an in-depth evaluation on the ruleset algorithm’s performance and feasibility of applying the generated rules on the game-induced EEG data for justifying the emotional state prediction result.

FetchEEG: a hybrid approach combining feature extraction and temporal-channel joint attention for EEG-based emotion classification

Objective. Electroencephalogram (EEG) analysis has always been an important tool in neural engineering, and the recognition and classification of human emotions are one of the important tasks in neural engineering. EEG data, obtained from electrodes placed on the scalp, represent a valuable resource of information for brain activity analysis and emotion recognition. Feature extraction methods have shown promising results, but recent trends have shifted toward end-to-end methods based on deep learning. However, these approaches often overlook channel representations, and their complex structures pose certain challenges to model fitting. Approach. To address these challenges, this paper proposes a hybrid approach named FetchEEG that combines feature extraction and temporal-channel joint attention. Leveraging the advantages of both traditional feature extraction and deep learning, the FetchEEG adopts a multi-head self-attention mechanism to extract representations between different time moments and channels simultaneously. The joint representations are then concatenated and classified using fully-connected layers for emotion recognition. The performance of the FetchEEG is verified by comparison experiments on a self-developed dataset and two public datasets. Main results. In both subject-dependent and subject-independent experiments, the FetchEEG demonstrates better performance and stronger generalization ability than the state-of-the-art methods on all datasets. Moreover, the performance of the FetchEEG is analyzed for different sliding window sizes and overlap rates in the feature extraction module. The sensitivity of emotion recognition is investigated for three- and five-frequency-band scenarios. Significance. FetchEEG is a novel hybrid method based on EEG for emotion classification, which combines EEG feature extraction with Transformer neural networks. It has achieved state-of-the-art performance on both self-developed datasets and multiple public datasets, with significantly higher training efficiency compared to end-to-end methods, demonstrating its effectiveness and feasibility.

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning.

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Human Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM

Human emotion recognition is crucial in various technological domains, reflecting our growing reliance on technology. Facial expressions play a vital role in conveying and preserving human emotions. While deep learning has been successful in recognizing emotions in video sequences, it struggles to effectively model spatio-temporal interactions and identify salient features, limiting its accuracy. This research paper proposed an innovative algorithm for facial expression recognition which combined a deep learning algorithm and dynamic texture methods. In the initial phase of this study, facial features were extracted using the Visual-Geometry-Group (VGG19) model and input into Long-Short-Term-Memory (LSTM) cells to capture spatio-temporal information. Additionally, the HOG-HOF descriptor was utilized to extract dynamic features from video sequences, capturing changes in facial appearance over time. Combining these models using the Multimodal-Compact-Bilinear (MCB) model resulted in an effective descriptor vector. This vector was then classified using a Support Vector Machine (SVM) classifier, chosen for its simpler interpretability compared to deep learning models. This choice facilitates better understanding of the decision-making process behind emotion classification. In the experimental phase, the fusion method outperformed existing state-of-the-art methods on the eNTERFACE05 database, with an improvement margin of approximately 1%. In summary, the proposed approach exhibited superior accuracy and robust detection capabilities.

Evaluation of Eye-Blinking Dynamics in Human Emotion Recognition Using Weighted Visibility Graph

Purpose: Designing an automated emotion recognition system using biosignals has become a hot and challenging issue in many fields, including human-computer interferences, robotics, and affective computing. Several algorithms have been proposed to characterize the internal and external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are habitually analyzed in a multi-modality system using classic statistical measures, and the evaluation of its dynamics has been neglected so far.
 Materials and Methods: This experiment intended to provide an innovative single-modality scheme for emotion classification using eye-blinking data. The dynamics of eye-blinking data have been characterized by weighted visibility graph-based indices. The extracted measures were then fed to the different classifiers, including support vector machine, decision tree, k-Nearest neighbor, Adaptive Boosting, and random subset to complete the process of classifying sad, happy, neutral, and fearful affective states. The scheme has been evaluated utilizing the available signals in the SEED-IV database.
 Results: The proposed framework provided significant performance in terms of recognition rates. The highest average recognition rates of > 90% were achieved using the decision tree.
 Conclusion: In brief, our results showed that eye-blinking data has the potential for emotion recognition. The present system can be extended for designing future affect recognition systems.

A software package for the investigation of spatio-temporal feature descriptors

At present time researchers investigate approaches to the automatic facial micro-expression analysis to implement the possibility of the recognition of true human emotions from the image sequence of his face. The modern technologies are based on the use of complex mathematical algorithms with many adjusted parameters; in particular, such algorithms are the algorithms of spatio-temporal feature descriptors. The scientists who are not technical specialists, e.g., psychologists or psychotherapists, when investigating these algorithms may encounter difficulties in writing the program code. Therefore, it became necessary to develop a special software package for the investigation of spatio-temporal feature descriptors. Aim of the work is development of a special software package for studying various spatial and temporal descriptors of facial micromovement features in video images A software package for the investigation of spatio-temporal feature descriptors with a graphical interface is developed. The software package makes it possible to investigate the influence of the selection of the descriptor (LBP-TOP, FHOOF or FHOFO) and its parameters on the micro-facial movement detection accuracy for different input datasets of human face image sequences. The developed software package for the investigation of spatio-temporal feature descriptors can be applied as a software tool for carrying out investigations in the facial micro-expression analysis field.

Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data

Human emotion recognition, crucial for interpersonal relations and human-building interaction, identifies emotions from various behavioral signals to improve user interactions. To enhance the performance of emotion recognition, this study proposed a novel model that fuses physiological, environmental, and personal data. A unique dataset was created via experiments conducted in an environmental chamber, and an emotion recognition model was subsequently developed using a multimodal fusion approach. The model transforms physiological data into 2D images to capture time series and spatial features and uniquely incorporates metadata, including environmental and personal data. The model’s generalizability was validated using a leave-one-sample-out approach. The result showed 31.6% reduction of error with a predicted area when physiological, environmental, and personal data were fused in the emotion recognition model, suggesting that incorporating various contextual factors beyond physiological changes, such as the surrounding environment and inherent or acquired individual traits, can significantly enhance the model’s understanding of emotions. Furthermore, the model was to be robust to individual differences, offering consistent emotion recognition across different subjects. These findings suggest that the proposed model can serve as a potent tool for emotion recognition in built environmental applications.

Cross subject emotion identification from multichannel EEG sub-bands using Tsallis entropy feature and KNN classifier

Human emotion recognition remains a challenging and prominent issue, situated at the convergence of diverse fields, such as brain–computer interfaces, neuroscience, and psychology. This study utilizes an EEG data set for investigating human emotion, presenting novel findings and a refined approach for EEG-based emotion detection. Tsallis entropy features, computed for q values of 2, 3, and 4, are extracted from signal bands, including theta-θ (4–7 Hz), alpha-α (8–15 Hz), beta-β (16–31 Hz), gamma-γ (32–55 Hz), and the overall frequency range (0–75 Hz). These Tsallis entropy features are employed to train and test a KNN classifier, aiming for accurate identification of two emotional states: positive and negative. In this study, the best average accuracy of 79% and an F-score of 0.81 were achieved in the gamma frequency range for the Tsallis parameter q = 3. In addition, the highest accuracy and F-score of 84% and 0.87 were observed. Notably, superior performance was noted in the anterior and left hemispheres compared to the posterior and right hemispheres in the context of emotion studies. The findings show that the proposed method exhibits enhanced performance, making it a highly competitive alternative to existing techniques. Furthermore, we identify and discuss the shortcomings of the proposed approach, offering valuable insights into potential avenues for improvements.