Emotion Classification System Research Articles

Emotion Classification System for ASD Group by Using Wireless EEG Monitoring Device

This paper delves into a study employing dry electroencephalograms (EEG) as input features to discern emotions in individuals with autism spectrum disorders (ASD). While EEG is a prevalent tool in emotion classification studies for typically developing individuals, less attention has been directed towards its application in the ASD population. In this study, ten participants diagnosed with ASD wore wireless dry EEG sensors to capture their EEG signals. These signals encompassed alpha, beta, delta, theta, and gamma waves, which were subsequently subjected to feature extraction techniques such as the t-test, principal component analysis (PCA), ReliefF, and Chi-Square. Classification of positive, neutral, and negative emotions was performed using various algorithms, including K-nearest neighbor (KNN), Multinomial Logistic Regression (MLR), Naive Bayes (NB), Random Tree (RT), Random Forest (RF), and Support Vector Machine (SVM). Ultimately, employing SVM with a t-test enhanced the accuracy of emotion classification for the ASD group from 66.4% to 74.1%.

Emotion-Aware In-Car Feedback: A Comparative Study

We investigate personalised feedback mechanisms to help drivers regulate their emotions, aiming to improve road safety. We systematically evaluate driver-preferred feedback modalities and their impact on emotional states. Using unobtrusive vision-based emotion detection and self-labeling, we captured the emotional states and feedback preferences of 21 participants in a simulated driving environment. Results show that in-car feedback systems effectively influence drivers’ emotional states, with participants reporting positive experiences and varying preferences based on their emotions. We also developed a machine learning classification system using facial marker data to demonstrate the feasibility of our approach for classifying emotional states. Our contributions include design guidelines for tailored feedback systems, a systematic analysis of user reactions across three feedback channels with variations, an emotion classification system, and a dataset with labeled face landmark annotations for future research.

Optimizing Hyperparameters of CNN and DNN for Emotion Classification Based on EEG Signals

EEG emotion is a research topic that has gained significant attention in the development of emotion classification systems. This study focuses on optimizing the hyperparameters of CNN (Convolutional Neural Network) and DNN (Deep Neural Network) for classifying EEG emotion signals. The data is divided into three train-test data ratio scenarios: 80:20, 70:30, and 60:40. After modeling and the classification process, hyperparameter tuning was conducted on both models to achieve the best results. Experimental results showed the highest accuracy of 98.36% for CNN, while DNN reached 98.18% in the 80:20 data ratio scenario. Despite the high accuracy, the differences in the loss curves between CNN and DNN reflect the complexity of the performance of both models. The train-test data ratio was also found to significantly impact the performance of both models, with the 80:20 data split yielding the best results, while the 70:30 and 60:40 splits resulted in slightly lower accuracies.

Emotion classification with multi‐modal physiological signals using multi‐attention‐based neural network

AbstractThe ability to effectively classify human emotion states is critically important for human‐computer or human‐robot interactions. However, emotion classification with physiological signals is still a challenging problem due to the diversity of emotion expression and the characteristic differences in different modal signals. A novel learning‐based network architecture is presented that can exploit four‐modal physiological signals, electrocardiogram, electrodermal activity, electromyography, and blood volume pulse, and make a classification of emotion states. It features two kinds of attention modules, feature‐level, and semantic‐level, which drive the network to focus on the information‐rich features by mimicking the human attention mechanism. The feature‐level attention module encodes the rich information of each physiological signal. While the semantic‐level attention module captures the semantic dependencies among modals. The performance of the designed network is evaluated with the open‐source Wearable Stress and Affect Detection dataset. The developed emotion classification system achieves an accuracy of 83.88%. Results demonstrated that the proposed network could effectively process four‐modal physiological signals and achieve high accuracy of emotion classification.

Students’ emotion classification system through an ensemble approach

Students’ emotion classification system through an ensemble approach

Emotion Fusion-Sense (Emo Fu-Sense) – A novel multimodal emotion classification technique

Human emotions play a vital role in overall well-being. With the advent of advance technologies growing interest has been observed in developing a multimodal emotion classification system that can accurately interpret human emotions. The article presents a comprehensive overview of a multimodal emotion classification system (Emo Fu-Sense) designed to capture the rich and nuanced nature of human emotions. Objective of Emo Fu-Sense is to integrates information from Electrocardiogram (ECG), Galvanic Skin Response (GSR), Electroencephalograph (EEG), respiration amplitude and body temperature to achieve holistic understanding of emotional states. To effectively extract information from multimodal data, designed system employs conventional methods with sophisticated machine learning algorithms including Long Short-Term Memory (LSTM), a variety of Recurrent Neural Network (RNN). Recommended solution extracts column wise features independently from different modalities based on the windowing operation. Finally, feature fusion and modality biasing were used to combine the information from different modalities. The proposed method has not only highlighted the limitations of unimodal system but has achieved a classification accuracy of 92.62 %, with an average F1-Score of 93 % and 9.2 % of Mean Absolute Error (MAE). Obtained results are better than existing state-of-the-art approaches. Evaluation of the multimodal emotion classification system was conducted on MAHNOB-HCI dataset, which encompasses a wide range of emotional expressions across various contexts and individuals. The integration of multiple modalities and advanced machine learning techniques enables a more comprehensive understanding of emotional states and highlights the significance of research and development in the field of affective computing.

Advancements in EEG Emotion Recognition: Leveraging Multi-Modal Database Integration

This paper addresses the limitations of relying solely on facial expressions for emotion recognition by proposing an advanced approach that emphasizes continuous monitoring of electroencephalography (EEG) signals. Recognizing the potential for deception in facial expressions, our study leverages the growing interest in EEG signals, tapping into advancements in deep learning and machine learning. By optimizing the configuration of EEG electrodes, our approach enhances the accuracy of emotion classification systems, offering a streamlined solution. The proposed multi-input system refines EEG-based emotion recognition efficiency and integrates facial expression analysis to enhance overall system effectiveness. Through the application of brain heat map topographies and facial expression recognition, our system, employing just nine electrodes, outperforms basic emotion recognition setups. Experimental results validate that combining facial expression analysis with EEG signals provides a more comprehensive and accurate understanding of human emotions. This innovative approach holds significance across various sectors, including healthcare, psychology, and human–computer interaction. The paper introduces a novel multi-input system approach, collaboratively fusing two powerful deep learning algorithms: two Convolutional Neural Networks (CNNs). The proposed EEG-based CNN algorithm achieves an efficiency of 87.43%, rising to 91.21% when integrated with the DeepFace CNN. The seamless integration of facial expressions and brain topographies enables the system to efficiently harness abundant information from both modalities, ensuring a thorough comprehension of human emotions. By capitalizing on the combined advantages of analyzing facial expressions and EEG-derived brain topography, this avant-garde technique substantially improves both precision and efficiency in emotion recognition systems. This enhancement establishes a foundation for the introduction of innovative applications across a spectrum of fields.

Music emotion classification using a hybrid CNN-LSTM model

The emotional content of music, interwoven with the intricacies of human affect, poses a unique challenge for computational recognition and classification. With the digitalization of music libraries expanding exponentially, there is a pressing need for precise, automated tools capable of navigating and categorizing vast musical repositories based on emotional contexts. This study advances music emotion classification in the field of music information retrieval by developing a deep learning model that accurately predicts emotional categories in music. The goal of this research is to advance the field of music emotion classification by leveraging the capabilities of convolutional neural networks combined with long short-term memory within deep learning frameworks. The contribution of this study is to provide a refined approach to music emotion classification, combining the power of convolutional neural networks and long short-term memory architectures with sophisticated preprocessing of the Emotify dataset for a deeper and more accurate analysis of musical emotions. The research introduces a novel architecture combining Convolutional Neural Networks and Long Short-Term Memory networks designed to capture the intricate emotional nuances in music. The model leverages convolutional neural networks for robust feature detection and Long Short-Term Memory networks for effective sequence learning, addressing the temporal dynamics of musical features. Utilizing the Emotify dataset, comprising tracks annotated with nine emotional features, the study expands the dataset by segmenting each track into 20 parts, thereby enriching the variety of emotional expressions. Techniques like the synthetic minority oversampling technique were implemented to counter dataset imbalance, ensuring equitable representation of various emotions. The spectral characteristics of the samples were analyzed using the Fast Fourier Transform, contributing to a more comprehensive understanding of the data. Through meticulous fine-tuning, including dropout implementation to prevent overfitting and learning rate adjustments, the developed model achieved a notable accuracy of 94.7 %. This high level of precision underscores the model's potential for application in digital music services, recommendation systems, and music therapy. Future enhancements to this music emotion classification system include expanding the dataset and refining the model architecture for even more nuanced emotional analysis

A Hybrid Deep Learning Emotion Classification System Using Multimodal Data.

This paper proposes a hybrid deep learning emotion classification system (HDECS), a hybrid multimodal deep learning system designed for emotion classification in a specific national language. Emotion classification is important in diverse fields, including tailored corporate services, AI advancement, and more. Additionally, most sentiment classification techniques in speaking situations are based on a single modality: voice, conversational text, vital signs, etc. However, analyzing these data presents challenges because of the variations in vocal intonation, text structures, and the impact of external stimuli on physiological signals. Korean poses challenges in natural language processing, including subject omission and spacing issues. To overcome these challenges and enhance emotion classification performance, this paper presents a case study using Korean multimodal data. The case study model involves retraining two pretrained models, LSTM and CNN, until their predictions on the entire dataset reach an agreement rate exceeding 0.75. Predictions are used to generate emotional sentences appended to script data, which are further processed using BERT for final emotion prediction. The research result is evaluated by using categorical cross-entropy (CCE) to measure the difference between the model's predictions and actual labels, F1 score, and accuracy. According to the evaluation, the case model outperforms the existing KLUE/roBERTa model with improvements of 0.5 in CCE, 0.09 in accuracy, and 0.11 in F1 score. As a result, the HDECS is expected to perform well not only on Korean multimodal datasets but also on sentiment classification considering the speech characteristics of various languages and regions.

Neurofinance: Exploratory Analysis Stock Trader's Decision-Making Process by Real-Time Monitoring of Emotional Reactions

Human emotions can be associated with decision-making, and emotions can generate behaviors. Due to the fact that it could be biased and exhaustively complex to examine how human beings make choices, it is necessary to consider relevant groups of study, such as stock traders and non-traders in finance. This work aims to analyze the connection between emotions and the decision-making process of investors and non-investors submitted to the same set of stimuli to understand how emotional arousal might dictate the decision process. Neuroscience monitoring tools such as Real-Time Facial Expression Analysis (AFFDEX), Eye-Tracking, and Galvanic Skin Response (GSR) were adopted to monitor the related experiments of this paper and its accompanying analysis process. Thirty-seven participants attended the study, 24 were classified as stock traders, and 13 were non-traders; the mean age for the groups was 35 and 25, respectively. The designed experiment initially disclosed a thought-provoking result between the two groups under the certainty and risk-seeking prospect theory; there were more risk-takers among non-investors at 75%, while investors were inclined toward certainty at 79.17%. The implication could be that the non-investing individuals were less complex in thought and therefore pursued higher returns besides a high probability of losing the game. In addition, the automatic emotion classification system indicates that when non-investors confronted a stock trending chart beyond their acquaintance or knowledge, they were psychologically exposed to fear, anger, sadness, and surprise. On the contrary, investors were detected with disgust, joy, contempt, engagement, sadness, and surprise, where sadness and surprise overlapped in both parties. Under time pressure conditions, 54.05% of investors or non-investors tend to make decisions after the peak(s) of emotional arousal. Variations were found in the deciding points of the slopes: 2.70% were decided right after the peak(s), 37.84% waited until the emotions turned stable, and 13.51% were determined as the emotional indicators started to slide downwards. Several combinations of emotional responses were associated with decisions. For example, negative emotions could induce passive decision-making, in this case, to sell the stock; nevertheless, it was also examined that as the slope slipped downwards to a particular horizontal point, the individuals became more optimistic and selected the "BUY" option. Future works may consider expanding the study to larger sample size, different demographic groups, and other biometrics for further analysis and conclusions.

Automatic detection of artifacts and improved classification models for emotional activity detection from multimodal physiological data

This manuscript proposes an automated artifacts detection and multimodal classification system for human emotion analysis from human physiological signals. First, multimodal physiological data, including the Electrodermal Activity (EDA), electrocardiogram (ECG), Blood Volume Pulse (BVP) and respiration rate signals are collected. Second, a Modified Compressed Sensing-based Decomposition (MCSD) is used to extract the informative Skin Conductance Response (SCR) events of the EDA signal. Third, raw features (edge and sharp variations), statistical and wavelet coefficient features of EDA, ECG, BVP, respiration and SCR signals are obtained. Fourth, the extracted raw features, statistical and wavelet coefficient features from all physiological signals are fed into the parallel Deep Convolutional Neural Network (DCNN) to reduce the dimensionality of feature space by removing artifacts. Fifth, the fused artifact-free feature vector is obtained for neutral, stress and pleasure emotion classes. Sixth, an artifact-free feature vector is used to train the Random Forest Deep Neural Network (RFDNN) classifier. Then, a trained RFDNN classifier is applied to classify the test signals into different emotion classes. Thus, leveraging the strengths of both RF and DNN algorithms, more comprehensive feature learning using multimodal psychological data is achieved, resulting in robust and accurate classification of human emotional activities. Finally, an extensive experiment using the Wearable Stress and Affect Detection (WESAD) dataset shows that the proposed system outperforms other existing human emotion classification systems using physiological data.

Estimating the Uncertainty in Emotion Class Labels With Utterance-Specific Dirichlet Priors

Emotion recognition is a key attribute for artificial intelligence systems that need to naturally interact with humans. However, the task definition is still an open problem due to the inherent ambiguity of emotions. In this paper, a novel Bayesian training loss based on per-utterance Dirichlet prior distributions is proposed for verbal emotion recognition, which models the uncertainty in one-hot labels created when human annotators assign the same utterance to different emotion classes. An additional metric is used to evaluate the performance by detection test utterances with high labelling uncertainty. This removes a major limitation that emotion classification systems only consider utterances with labels where the majority of annotators agree on the emotion class. Furthermore, a frequentist approach is studied to leverage the continuous-valued "soft" labels obtained by averaging the one-hot labels. We propose a two-branch model structure for emotion classification on a per-utterance basis, which achieves state-of-the-art classification results on the widely used IEMOCAP dataset. Based on this, uncertainty estimation experiments were performed. The best performance in terms of the area under the precision-recall curve when detecting utterances with high uncertainty was achieved by interpolating the Bayesian training loss with the Kullback-Leibler divergence training loss for the soft labels. The generality of the proposed approach was verified using the MSP-Podcast dataset which yielded the same pattern of results.

Comparison of Various Feature Selection Algorithms in Speech Emotion Recognition

Speech Emotion Recognition (SER) plays a predominant role in human-machine interaction. SER is a challenging task because of number of complexities involved in it. For an accurate emotion classification system, feature extraction is the first and important step carried out on speech signals. And after the features are extracted, it is very important to select the best features out of all and reject the redundant and least important features. Feature selection methods play an important role in SER performance. The classifier gets the selected features, so as to reduce the unnecessary overload and perform better to classify the emotions. In this study, a good combination of features is selected from Punjabi Emotional Speech Database. Then a number of feature selection algorithms are explored and experimented upon, to select the best features. 1D-CNN is used for classification purpose. The results are shown and compared on the basis of number of performance metrics. LASSO has shown the best performance results as compared to other feature selection methods.

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions.

Multimodal emotion recognition has gained much traction in the field of affective computing, human-computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.

Instrumental Music Emotion Recognition with MFCC and KNN Algorithm

Every piece of music contains emotion in every sound presented. Detection of the music emotion is quite difficult to do because the emotions felt are subjective. Based on this problem, it is necessary to have an automatic classification system to detect the emotions produced in music. In this paper, an explanation of the result to develop an emotional classification system of instrumental music. This system described the process starting with the receiving an input in the form of a music file in the format wav. Furthermore, the feature extraction process is carried out using Mel-Frequency Cepstral Coefficients (MFCC). The result of the extraction of such features are used in the classification process using the K-Nearest Neighbor (K-NN). The system produced output in the form of happy, relaxed, and sad emotions. The output of the system has a classification achieved an accuracy of 97.5% for the value of k = 1, reaching an accuracy of 95% for the value of k = 2.95% and for k = 3, reaching an accuracy of up to 90%.

China public emotion analysis under normalization of COVID-19 epidemic: Using Sina Weibo.

The prevention and control of the coronavirus disease 2019 (COVID-19) epidemic in China has entered a phase of normalization. The basis for evaluating and improving public health strategies is understanding the emotions and concerns of the public. This study establishes a fine-grained emotion-classification model to annotate the emotions of 32,698 Sina Weibo posts related to COVID-19 prevention and control from July 2022 to August 2022. The Dalian University of Technology (DLUT) emotion-classification system was adjusted to form four pairs (eight categories) of bidirectional emotions: good-disgust, joy-sadness, anger-fear, and surprise-anticipation. A lexicon-based method was proposed to classify the emotions of Weibo posts. Based on the selected Weibo posts, the present study analyzed the Chinese public's sentiments and emotions. The results showed that positive sentiment accounted for 51%, negative sentiment accounted for 24%, and neutral sentiment accounted for 25%. Positive sentiments were dominated by good and joy emotions, and negative sentiments were dominated by fear and disgust emotions. The proportion of positive sentiments on official Weibo (accounts belonging to government departments and official media) is significantly higher than that on personal Weibo. Official Weibo users displayed a weak guiding effect on personal users in terms of positive sentiment and the two groups of users were almost completely synchronized in terms of negative sentiment. The linear discriminant analysis (LDA) was performed on the two negative emotions of fear and disgust in the personal posts. The present study found that the emotion of fear was mainly related to COVID-19 infection and death, control of people with positive nucleic acid tests, and the outbreak of local epidemic, while the emotion of disgust was mainly related to the long-term existence of the epidemic, the cost of nucleic acid tests, non-implementation of prevention and control measures, and the occurrence of foreign epidemics. These findings suggest that Chinese attitudes toward epidemic prevention and control are positive and optimistic; however, there is also a notable proportion of fear and disgust. It is expected that this study will help public health administrators to evaluate the effectiveness of possible countermeasures and work toward precise prevention and control of the COVID-19 epidemic.

Proposal for an emotion classification system from dog’s tail movements using the information on the surrounding environment

Proposal for an emotion classification system from dog’s tail movements using the information on the surrounding environment

Automated robust human emotion classification system using hybrid EEG features with ICBrainDB dataset.

Emotion identification is an essential task for human-computer interaction systems. Electroencephalogram (EEG) signals have been widely used in emotion recognition. So far, there have been several EEG-based emotion recognition datasets that the researchers have used to validate their developed models. Hence, we have used a new ICBrainDB EEG dataset to classify angry, neutral, happy, and sad emotions in this work. Signal processing-based wavelet transform (WT), tunable Q-factor wavelet transform (TQWT), and image processing-based histogram of oriented gradients (HOG), local binary pattern (LBP), and convolutional neural network (CNN) features have been used extracted from the EEG signals. The WT is used to extract the rhythms from each channel of the EEG signal. The instantaneous frequency and spectral entropy are computed from each EEG rhythm signal. The average, and standard deviation of instantaneous frequency, and spectral entropy of each rhythm of the signal are the final feature vectors. The spectral entropy in each channel of the EEG signal after performing the TQWT is used to create the feature vectors in the second signal side method. Each EEG channel is transformed into time-frequency plots using the synchrosqueezed wavelet transform. Then, the feature vectors are constructed individually using windowed HOG and LBP features. Also, each channel of the EEG data is fed to a pretrained CNN to extract the features. In the feature selection process, the ReliefF feature selector is employed. Various feature classification algorithms namely, k-nearest neighbor (KNN), support vector machines, and neural networks are used for the automated classification of angry, neutral, happy, and sad emotions. Our developed model obtained an average accuracy of 90.7% using HOG features and a KNN classifier with a tenfold cross-validation strategy.

Study on Music Emotion Recognition Based on the Machine Learning Model Clustering Algorithm

In recent years, the explosive growth of online music resources makes it difficult to retrieve and manage music information. To efficiently retrieve and classify music information has become a hot research topic. Thayer’s two-dimensional emotion plane is selected as the basis for establishing the music emotion database. Music is divided into five categories, the concept of continuous emotion perception is introduced, and music emotion is regarded as a point on a two-dimensional emotional plane, together with the two sentiment variables to determine its location. The artificial labeling method is used to determine the position range of the five types of emotions on the emotional plane, and the regression method is used to obtain the relationship between the VA value and the music features so that the music emotion classification problem is transformed into a regression problem. A regression-based music emotion classification system is designed and implemented, which mainly includes a training part and a testing part. In the training part, three algorithms, namely, polynomial regression, support vector regression, and k-plane piecewise regression, are used to obtain the regression model. In the test part, the input music data is regressed and predicted to obtain its VA value and then classified, and the system performance is considered by classification accuracy. Results show that the combined method of support vector regression and k-plane piecewise regression improves the accuracy by 3 to 4 percentage points compared to using one algorithm alone; compared with the traditional classification method based on a support vector machine, the accuracy improves by 6 percentage points. Music emotion is classified by algorithms such as support vector machine classification, K-neighborhood classification, fuzzy neural network classification, fuzzy K-neighborhood classification, Bayesian classification, and Fisher linear discrimination, among which the support vector machine, fuzzy K-neighborhood, and the accuracy rate of music emotion classification realized by Fisher linear discriminant algorithm are more than 80%; a new algorithm “mixed classifier” is proposed, and the music emotion recognition rate based on this algorithm reaches 84.9%.

Comparing Approaches for Explaining DNN-Based Facial Expression Classifications

Classifying facial expressions is a vital part of developing systems capable of aptly interacting with users. In this field, the use of deep-learning models has become the standard. However, the inner workings of these models are unintelligible, which is an important issue when deploying them to high-stakes environments. Recent efforts to generate explanations for emotion classification systems have been focused on this type of models. In this work, an alternative way of explaining the decisions of a more conventional model based on geometric features is presented. We develop a geometric-features-based deep neural network (DNN) and a convolutional neural network (CNN). Ensuring a sufficient level of predictive accuracy, we analyze explainability using both objective quantitative criteria and a user study. Results indicate that the fidelity and accuracy scores of the explanations approximate the DNN well. From the performed user study, it becomes clear that the explanations increase the understanding of the DNN and that they are preferred over the explanations for the CNN, which are more commonly used. All scripts used in the study are publicly available.