Classification Of Emotional States Research Articles

Riding Feeling Recognition Based on Multi-Head Self-Attention LSTM for Driverless Automobile

With the emergence of driverless technology, passenger ride comfort has become an issue of concern. In recent years, driving fatigue detection and braking sensation evaluation based on EEG signals have received more attention, and analyzing ride comfort using EEG signals is also a more intuitive method. However, it is still a challenge to find an effective method or model to evaluate passenger comfort. In this paper, we propose a long- and short-term memory network model based on a multiple self-attention mechanism for passenger comfort detection. By applying the multiple attention mechanism to the feature extraction process, more efficient classification results are obtained. The results show that the long- and short-term memory network using the multi-head self-attention mechanism is efficient in decision making along with higher classification accuracy. In conclusion, the classifier based on the multi-head attention mechanism proposed in this paper has excellent performance in EEG classification of different emotional states, and has a broad development prospect in brain-computer interaction.

Hybrid deep models for parallel feature extraction and enhanced emotion state classification

Emotions play a vital role in recognizing a person’s thoughts and vary significantly with stress levels. Emotion and stress classification have gained considerable attention in robotics and artificial intelligence applications. While numerous methods based on machine learning techniques provide average classification performance, recent deep learning approaches offer enhanced results. This research presents a hybrid deep learning model that extracts features using AlexNet and DenseNet models, followed by feature fusion and dimensionality reduction via Principal Component Analysis (PCA). The reduced features are then classified using a multi-class Support Vector Machine (SVM) to categorize different types of emotions. The proposed model was evaluated using the DEAP and EEG Brainwave datasets, both well-suited for emotion analysis due to their comprehensive EEG signal recordings and diverse emotional stimuli. The DEAP dataset includes EEG signals from 32 participants who watched 40 one-minute music videos, while the EEG Brainwave dataset categorizes emotions into positive, negative, and neutral based on EEG recordings from participants exposed to six different film clips. The proposed model achieved an accuracy of 95.54% and 97.26% for valence and arousal categories in the DEAP dataset, respectively, and 98.42% for the EEG Brainwave dataset. These results significantly outperform existing methods, demonstrating the model’s superior performance in terms of precision, recall, F1-score, specificity, and Mathew correlation coefficient. The integration of AlexNet and DenseNet, combined with PCA and multi-class SVM, makes this approach particularly effective for capturing the intricate patterns in EEG data, highlighting its potential for applications in human-computer interaction and mental health monitoring, marking a significant advancement over traditional methods.

Emotion classification using electrocardiogram and machine learning: A study on the effect of windowing techniques

Automated emotion recognition using physiological signals has gained significant attention in recent years due to its potential applications in human–computer interaction, healthcare, and psychology. Electrocardiogram (ECG) signals are widely used due to their non-invasiveness, high temporal resolution, and direct relationship with the autonomic nervous system. In this study, we propose a novel approach for ECG-based emotion recognition using time-series to image encoding techniques and texture-based features combined with machine learning algorithms and deep learning architectures. The ECG data used in this study were obtained from the Continuously Annotated Signals of Emotion (CASE) and Wearable Stress and Affect Detection (WESAD) datasets. We categorized emotional states based on valence and arousal annotations into four classes: High-Valence High-Arousal, High-Valence Low-Arousal, Low-Valence High-Arousal, and Low-Valence Low-Arousal. The ECGs were segmented into 5 and 7-window segments and transformed into 2D representations using Gramian Angular Summation Field, Markov Transition Field, Recurrence Plot (RP), and the triple-channel fusion of all these images. Total of 85 textural features based on the Gray-Level Co-occurrence Matrix (GLCM), Gray-Level Run Length Matrix, Zernike’s Moments, Hu’s Moments, Fractal Dimension Texture Analysis (FDTA), and First-Order Statistics were extracted. Classifiers, namely Random Forest, Support Vector Machine (SVM), eXtreme Gradient Boosting (XGB), 1D Convolutional Neural Network (CNN), and Multi-head Attention Network were considered to classify the emotional states. The performance of the classifiers varied depending on the time-series to image encoding technique, segmentation approaches, and the classifier employed. We achieved the highest Weighted F-measure (F-m) of 94.91% (RP + XGB) and 86.78% (RP + SVM) using the 7-window and 5-window approaches, respectively. Our proposed 1D CNN architecture achieved the highest classification metrics (F-m = 92.52%, Balanced accuracy = 92.0%, Recall = 91.96%, and Precision = 93.16%) with RP images in a 7-window approach. GLCM and FDTA features made significant contributions to the classification of emotional states. Overall, our results suggest that the proposed method holds promise for developing more accurate and efficient emotion recognition systems.

AFLEMP: Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data

Automated emotion recognition systems utilizing physiological signals are essential for affective computing and intelligent interaction. Combining the multiple physiological signals is more precise and effective in accurately assessing a person’s emotional state. These automated emotion recognition systems using conventional machine learning techniques require complete access to the physiological data for emotion state classification, compromising sensitive data privacy. Federated Learning (FL) resolves this issue by preserving the user’s privacy and sensitive physiological data while recognizing emotions. However, existing FL methods have limitations in handling data heterogeneity in the physiological data and do not measure communication efficiency and scalability. In response to these challenges, this paper proposes a unique novel framework called AFLEMP (Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data) integrating attention mechanism-based Transformer with an Artificial Neural Network (ANN) model. The framework reduces two types of data heterogeneity: (1) Variation Heterogeneity (VH) in multi-modal EEG, GSR, and ECG physiological signal data using attention mechanisms and (2) Imbalanced Data Heterogeneity (IDH) in the FL environment using scaled weighted federated averaging. This paper validates the proposed AFLEMP framework on two publicly available emotion datasets, AMIGOS and DREAMER, achieving an average accuracy of 88.30% and 84.10%, respectively. The proposed AFLEMP framework proves robust, scalable, and efficient in communication. AFLEMP is the first FL framework to propose for emotion recognition using multi-modal physiological signals while reducing data heterogeneity and outperforming existing FL methods.

Neural correlates of recalled sadness, joy, and fear states: a source reconstruction EEG study.

The capacity to understand the others' emotional states, particularly if negative (e.g. sadness or fear), underpins the empathic and social brain. Patients who cannot express their emotional states experience social isolation and loneliness, exacerbating distress. We investigated the feasibility of detecting non-invasive scalp-recorded electrophysiological signals that correspond to recalled emotional states of sadness, fear, and joy for potential classification. The neural activation patterns of 20 healthy and right-handed participants were studied using an electrophysiological technique. Analyses were focused on the N400 component of Event-related potentials (ERPs) recorded during silent recall of subjective emotional states; Standardized weighted Low-resolution Electro-magnetic Tomography (swLORETA) was employed for source reconstruction. The study classified individual patterns of brain activation linked to the recollection of three distinct emotional states into seven regions of interest (ROIs). Statistical analysis (ANOVA) of the individual magnitude values revealed the existence of a common emotional circuit, as well as distinct brain areas that were specifically active during recalled sad, happy and fearful states. In particular, the right temporal and left superior frontal areas were more active for sadness, the left limbic region for fear, and the right orbitofrontal cortex for happy affective states. In conclusion, this study successfully demonstrated the feasibility of detecting scalp-recorded electrophysiological signals corresponding to internal and subjective affective states. These findings contribute to our understanding of the emotional brain, and have potential applications for future BCI classification and identification of emotional states in LIS patients who may be unable to express their emotions, thus helping to alleviate social isolation and sense of loneliness.

Role of machine learning and deep learning techniques in EEG-based BCI emotion recognition system: a review

Emotion is a subjective psychophysiological reaction coming from external stimuli which impacts every aspect of our daily lives. Due to the continuing development of non-invasive and portable sensor technologies, such as brain-computer interfaces (BCI), intellectuals from several fields have been interested in emotion recognition techniques. Human emotions can be recognised using a variety of behavioural cues, including gestures and body language, voice, and physiological markers. The first three, however, might be ineffective because people sometimes conceal their genuine emotions either intentionally or unknowingly. More precise and objective emotion recognition can be accomplished using physiological signals. Among other physiological signals, Electroencephalogram (EEG) is more responsive and sensitive to variation in affective states. Various EEG-based emotion recognition methods have recently been introduced. This study reviews EEG-based BCIs for emotion identification and gives an outline of the progress made in this field. A summary of the datasets and techniques utilised to evoke human emotions and various emotion models is also given. We discuss several EEG feature extractions, feature selection/reduction, machine learning, and deep learning algorithms in accordance with standard emotional identification process. We provide an overview of the human brain's EEG rhythms, which are closely related to emotional states. We also go over a number of EEG-based emotion identification research and compare numerous machine learning and deep learning techniques. In conclusion, this study highlights the applications, challenges and potential areas for future research in identification and classification of human emotional states.

Investigation of different ML approaches in classification of emotions induced by acute stress

BackgroundMachine learning is becoming a common tool in monitoring emotion. However, methodological studies of the processing pipeline are scarce, especially ones using subjective appraisals as ground truth. New methodA novel protocol was used to induce cognitive load and physical discomfort, and emotional dimensions (arousal, valence, and dominance) were reported after each task. The performance of five common ML models with a versatile set of features (physiological features, task performance data, and personality trait) was compared in binary classification of subjectively assessed emotions. ResultsThe psychophysiological responses proved the protocol was successful in changing the mental state from baseline, also the cognitive and physical tasks were different. The optimization and performance of ML models used for emotion detection were evaluated. Additionally, methods to account for imbalanced classes were applied and shown to improve the classification performance. Comparison with existing method(s)Classification of human emotional states often assumes the states are determined by the stimuli. However, individual appraisals vary. None of the past studies have classified subjective emotional dimensions with a set of features including biosignals, personality and behavior. ConclusionOur data represent a typical setup in affective computing utilizing psychophysiological monitoring: N is low compared to number of features, inter-individual variability is high, and class imbalance cannot be avoided. Our observations are a) if possible, include features representing physiology, behavior and personality, b) use simple models and limited number of features to improve interpretability, c) address the possible imbalance, d) if the data size allows, use nested cross-validation.

Building a Database with Thermal Images for the Classification of Emotional States

Abstract This research paper aims primarily to highlight the stages regarding the development of a database with facial images in the thermal spectrum, presenting various emotional states (for example, neutral, happy and sad). A series of exploratory results was presented, carried out on five databases with thermal images (DBold-2019, DBnew-2023, DBnew-2023_Crop, DBnew- 2023_Noise and DBnew-2023_ROI), belonging to the CCETIC research center (CCETIC, 2023). The database development process includes both the thermal image acquisition part and the pre-processing steps (cutting, alignment, selection of areas of interest, removal of annotations/marking elements, etc). And secondly, it aims to present an analysis regarding the general structure of the thermal image processing application, together with a series of image processing methods and techniques, addressed in certain research works. Some experimental results obtained with the help of the five databases are presented, as well as a series of general conclusions regarding the development of a database with facial thermal images.

Classification of multiple emotional states from facial expressions in head-fixed mice using a deep learning-based image analysis.

Facial expressions are widely recognized as universal indicators of underlying internal states in most species of animals, thereby presenting as a non-invasive measure for assessing physical and mental conditions. Despite the advancement of artificial intelligence-assisted tools for automated analysis of voluminous facial expression data in human subjects, the corresponding tools for mice still remain limited so far. Considering that mice are the most prevalent model animals for studying human health and diseases, a comprehensive characterization of emotion-dependent patterns of facial expressions in mice could extend our knowledge on the basis of emotions and the related disorders. Here, we present a framework for the development of a deep learning-powered tool for classifying facial expressions in head-fixed mouse. We demonstrate that our machine vision was capable of accurately classifying three different emotional states from lateral facial images in head-fixed mouse. Moreover, we objectively determined how our classifier characterized the differences among the facial images through the use of an interpretation technique called Gradient-weighted Class Activation Mapping. Importantly, our machine vision presumably discerned the data by leveraging multiple facial features. Our approach is likely to facilitate the non-invasive decoding of a variety of emotions from facial images in head-fixed mice.

ELECTRODERMAL ACTIVITY-BASED ANALYSIS OF EMOTION RECOGNITION USING TEMPORAL-MORPHOLOGICAL FEATURES AND MACHINE LEARNING ALGORITHMS

In this study, we evaluated the performance of tonic and phasic components of Electrodermal activity (EDA) using machine learning algorithms for accurately recognizing emotions. The EDA signals considered for this study were obtained from Continuously Annotated Signals of Emotion (CASE) dataset. Initially, we pre-processed and decomposed the EDA into tonic and phasic components using cvxEDA method. Further, we extracted the temporal and morphological features from both tonic and phasic. Finally, we tested the performance of various combinations of features using machine learning algorithms such as logistic regression, support vector machine (SVM), and random forest. Our results revealed that the tonic contributes significant information for emotional state classification. Further, the temporal features of the phasic were able to discriminate most of the emotions [Formula: see text]. In particular, the scary emotion was well discriminated against other emotions. Results of classification revealed that SVM performed best in classifying emotional states. The results of our process pipeline, which incorporated tonic, temporal features, and SVM, showed impressive classification performance with average accuracy, sensitivity, specificity, precision, and f1-score of 78.96%, 57.92%, 85.97%, 62.32%, and 56.48%, respectively. Our findings indicate that our proposed models could potentially be used to detect the positive and negative emotions in healthcare settings.

Electroencephalography based human emotion state classification using principal component analysis and artificial neural network

In recent decades, the automatic emotion state classification is an important technology for human-machine interactions. In Electroencephalography (EEG) based emotion classification, most of the existing methodologies cannot capture the context information of the EEG signal and ignore the correlation information between dissimilar EEG channels. Therefore, in this study, a deep learning based automatic method is proposed for effective emotion state classification. Firstly, the EEG signals were acquired from the real time and databases for emotion analysis using physiological signals (DEAP), and further, the band-pass filter from 0.3 Hz to 45 Hz is utilized to eliminate both high and low-frequency noise. Next, two feature extraction techniques power spectral density and differential entropy were employed for extracting active feature values, which effectively learn the contextual and spatial information of EEG signals. Finally, principal component analysis and artificial neural network were developed for feature dimensionality reduction and emotion state classification. The experimental evaluation showed that the proposed method achieved 96.38% and 97.36% of accuracy on DEAP, and 92.33% and 89.37% of accuracy on a real-time database for arousal and valence emotion states. The achieved recognition accuracy is higher compared to the support vector machine on both databases.

Emotional State Classification from MUSIC-Based Features of Multichannel EEG Signals.

Electroencephalogram (EEG)-based emotion recognition is a computationally challenging issue in the field of medical data science that has interesting applications in cognitive state disclosure. Generally, EEG signals are classified from frequency-based features that are often extracted using non-parametric models such as Welch's power spectral density (PSD). These non-parametric methods are not computationally sound due to having complexity and extended run time. The main purpose of this work is to apply the multiple signal classification (MUSIC) model, a parametric-based frequency-spectrum-estimation technique to extract features from multichannel EEG signals for emotional state classification from the SEED dataset. The main challenge of using MUSIC in EEG feature extraction is to tune its parameters for getting the discriminative features from different classes, which is a significant contribution of this work. Another contribution is to show some flaws of this dataset for the first time that contributed to achieving high classification accuracy in previous research works. This work used MUSIC features to classify three emotional states and achieve 97% accuracy on average using an artificial neural network. The proposed MUSIC model optimizes a 95-96% run time compared with the conventional classical non-parametric technique (Welch's PSD) for feature extraction.

A Brain Network Analysis-Based Double Way Deep Neural Network for Emotion Recognition.

Constructing reliable and effective models to recognize human emotional states has become an important issue in recent years. In this article, we propose a double way deep residual neural network combined with brain network analysis, which enables the classification of multiple emotional states. To begin with, we transform the emotional EEG signals into five frequency bands by wavelet transform and construct brain networks by inter-channel correlation coefficients. These brain networks are then fed into a subsequent deep neural network block which contains several modules with residual connection and enhanced by channel attention mechanism and spatial attention mechanism. In the second way of the model, we feed the emotional EEG signals directly into another deep neural network block to extract temporal features. At the end of the two ways, the features are concatenated for classification. To verify the effectiveness of our proposed model, we carried out a series of experiments to collect emotional EEG from eight subjects. The average accuracy of the proposed model on our emotional dataset is 94.57%. In addition, the evaluation results on public databases SEED and SEED-IV are 94.55% and 78.91%, respectively, demonstrating the superiority of our model in emotion recognition tasks.

A Comprehensive Analysis of the Success of Classification Algorithms for the Classification of Emotional States Based on the User’s Behavioral Characteristics

The contribution measures the possibilities of classifying the emotional state of the user based on the behavior when working with a computer and with a smartphone. It complements the results of the original work of the authors, which dealt with the development of the Emotnizer application, data collection and their processing and classification of the 4 basic emotions found simultaneously in Ekman and Russell’s emotional model. The application has been completely redesigned and currently allows the classification of 4 basic dimensions (anger, joy, sadness and relaxation), while within these dimensions it is able to recognize and classify 20 different emotional states based on valence and arousal. In the article, we compare the results achieved by us (achieved by the 3-fold cross-validation method) with the results of the authors’ most similar works from the analysis of the current state. We achieve the highest classification accuracy of up to 87.88% by classifying arousal (low, medium, high) with a neural network algorithm for the second data set. Among the classification results for valence (negative, neutral, positive), we proposed a neural network classifier that works with an accuracy of 85.61%.

Machine Learning Models for Classification of Human Emotions Using Multivariate Brain Signals

Humans can portray different expressions contrary to their emotional state of mind. Therefore, it is difficult to judge humans’ real emotional state simply by judging their physical appearance. Although researchers are working on facial expressions analysis, voice recognition, and gesture recognition; the accuracy levels of such analysis are much less and the results are not reliable. Hence, it becomes vital to have realistic emotion detector. Electroencephalogram (EEG) signals remain neutral to the external appearance and behavior of the human and help in ensuring accurate analysis of the state of mind. The EEG signals from various electrodes in different scalp regions are studied for performance. Hence, EEG has gained attention over time to obtain accurate results for the classification of emotional states in human beings for human–machine interaction as well as to design a program where an individual could perform a self-analysis of his emotional state. In the proposed scheme, we extract power spectral densities of multivariate EEG signals from different sections of the brain. From the extracted power spectral density (PSD), the features which provide a better feature for classification are selected and classified using long short-term memory (LSTM) and bi-directional long short-term memory (Bi-LSTM). The 2-D emotion model considered for the classification of frontal, parietal, temporal, and occipital is studied. The region-based classification is performed by considering positive and negative emotions. The performance accuracy of our previous model’s results of artificial neural network (ANN), support vector machine (SVM), K-nearest neighbor (K-NN), and LSTM was compared and 94.95% accuracy was received using Bi-LSTM considering four prefrontal electrodes.

Case study of features extraction and real time classification of emotion from speech on the basis with using neural nets

The paper deals with the issue of classification of emotional state from speech. Due to the applied k-NN algorithm, the original solution achieved an overall classification success in the range of 20 to 35%, depending on the used audio sample input data database. In the original application, we have used the Praat program to extract the characteristics. In the current version of the application, the use of Praat has been eliminated and we have developed our solution based on neural networks. Therefore, 3 experiments with forward, 1 and 2D convolutional neural networks were performed to determine the overall success of the classification. Their common feature is that the prediction success was always highest in tests with a test subset of the RAVDESS database, with the best result being obtained using a 1D convolutional network (78.93%). Tests with the EMO-DB database were successful at 35.76%, 31.75% and 25.49%. In all three experiments, the worst results were obtained in tests with the SAVEE database - 20.24%, 18.45% and 22.02%.

Reward-Penalty Weighted Ensemble for Emotion State Classification from Multi-Modal Data Streams.

Researchers have shown the limitations of using the single-modal data stream for emotion classification. Multi-modal data streams are therefore deemed necessary to improve the accuracy and performance of online emotion classifiers. An online decision ensemble is a widely used approach to classify emotions in real-time using multi-modal data streams. There is a plethora of online ensemble approaches; these approaches use a fixed parameter ([Formula: see text]) to adjust the weights of each classifier (called penalty) in case of wrong classification and no reward for a good performing classifier. Also, the performance of the ensemble depends on the [Formula: see text], which is set using trial and error. This paper presents a new Reward-Penalty-based Weighted Ensemble (RPWE) for real-time multi-modal emotion classification using multi-modal physiological data streams. The proposed RPWE is thoroughly tested using two prevalent benchmark data sets, DEAP and AMIGOS. The first experiment confirms the impact of the base stream classifier with RPWE for emotion classification in real-time. The RPWE is compared with different popular and widely used online ensemble approaches using multi-modal data streams in the second experiment. The average balanced accuracy, F1-score results showed the usefulness and robustness of RPWE in emotion classification in real-time from the multi-modal data stream.

A federated learning method for real-time emotion state classification from multi-modal streaming.

Emotional and physical health are strongly connected and should be taken care of simultaneously to ensure completely healthy persons. A person's emotional health can be determined by detecting emotional states from various physiological measurements (EDA, RB, EEG, etc.). Affective Computing has become the field of interest, which uses software and hardware to detect emotional states. In the IoT era, wearable sensor-based real-time multi-modal emotion state classification has become one of the hottest topics. In such setting, a data stream is generated from wearable-sensor devices, data accessibility is restricted to those devices only and usually a high data generation rate should be processed to achieve real-time emotion state responses. Additionally, protecting the users' data privacy makes the processing of such data even more challenging. Traditional classifiers have limitations to achieve high accuracy of emotional state detection under demanding requirements of decentralized data and protecting users' privacy of sensitive information as such classifiers need to see all data. Here comes the federated learning, whose main idea is to create a global classifier without accessing the users' local data. Therefore, we have developed a federated learning framework for real-time emotion state classification using multi-modal physiological data streams from wearable sensors, called Fed-ReMECS. The main findings of our Fed-ReMECS framework are the development of an efficient and scalable real-time emotion classification system from distributed multimodal physiological data streams, where the global classifier is built without accessing (privacy protection) the users' data in an IoT environment. The experimental study is conducted using the popularly used multi-modal benchmark DEAP dataset for emotion classification. The results show the effectiveness of our developed approach in terms of accuracy, efficiency, scalability and users' data privacy protection.

Early detection of cyber-bullying from twitter data

The vast volume of user created data in social media has created a wide-open path to data analytics in terms of extraction of features, analysis of polarity of information and classification of human emotional state. Sentiment analysis and data mining has reached at its peak in terms of its use in feature extraction and commercial analysis of products. Enormous approaches have been also seen in human behaviour analysis using ML (Machine Learning) algorithms and NLP (Natural Language Processing). To be very specific, it is not too hard to analyse data in granular level such as aspects. Social media play a distinct role in providing large dataset at no cost. And aspect-based sentiment analysis provides us the text analysis technique that categorisesuser provided data and assigns positive or negative value for the sentiment identified in it. In our approach we have tried to find out a solution of detection of cyber bullying in human interaction. This problem, we find a new threat in this digital era as commenting and replying to that comment play an important role in virtual communication, and just in this point, it takes no cost or hesitation factor to attack another person verbally or virtually.

Emotion Recognition With Knowledge Graph Based on Electrodermal Activity.

Electrodermal activity (EDA) sensor is emerging non-invasive equipment in affect detection research, which is used to measure electrical activities of the skin. Knowledge graphs are an effective way to learn representation from data. However, few studies analyzed the effect of knowledge-related graph features with physiological signals when subjects are in non-similar mental states. In this paper, we propose a model using deep learning techniques to classify the emotional responses of individuals acquired from physiological datasets. We aim to improve the execution of emotion recognition based on EDA signals. The proposed framework is based on observed gender and age information as embedding feature vectors. We also extract time and frequency EDA features in line with cognitive studies. We then introduce a sophisticated weighted feature fusion method that combines knowledge embedding feature vectors and statistical feature (SF) vectors for emotional state classification. We finally utilize deep neural networks to optimize our approach. Results obtained indicated that the correct combination of Gender-Age Relation Graph (GARG) and SF vectors improve the performance of the valence-arousal emotion recognition system by 4 and 5% on PAFEW and 3 and 2% on DEAP datasets.