Emotion Recognition Model Research Articles

S[formula omitted]T-Net: A novel electroencephalogram signals-oriented emotion recognition model

In this paper, a novel skipping spatial–spectral–temporal network (S3T-Net) is developed to handle intra-individual differences in electroencephalogram (EEG) signals for accurate, robust, and generalized emotion recognition. In particular, aiming at the 4D features extracted from the raw EEG signals, a multi-branch architecture is proposed to learn spatial–spectral cross-domain representations, which benefits enhancing the model generalization ability. Time dependency among different spatial–spectral features is further captured via a bi-directional long-short term memory module, which employs an attention mechanism to integrate context information. Moreover, a skip-change unit is designed to add another auxiliary pathway for updating model parameters, which alleviates the vanishing gradient problem in complex spatial–temporal network. Evaluation results show that the proposed S3T-Net outperforms other advanced models in terms of the emotion recognition accuracy, which yields an performance improvement of 0.23% , 0.13%, and 0.43% as compared to the sub-optimal model in three test scenes, respectively. In addition, the effectiveness and superiority of the key components of S3T-Net are demonstrated from various experiments. As a reliable and competent emotion recognition model, the proposed S3T-Net contributes to the development of intelligent sentiment analysis in human–computer interaction (HCI) realm.

Balancing performance and efficiency: A CNN-LSTM hybrid model for sentiment analysis

In the era of rapid e-commerce and social networks, the recognition of textual emotion has garnered escalating interest in recent years. This area of study proves pivotal for applications such as content recommendation and human-robot interaction. Prominent models such as CNN, LSTM, and Roberta have been used in natural language processing to decipher the subtleties of textual emotion. Despite the commendable accuracy achieved by Roberta, its extensive model parameters come at the cost of sluggish training speeds. To address this issue, we propose a novel hybrid approach that integrates Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks. We employ this hybrid model to address the emotion classification problem using a dataset named "Emotion", and we compare its performance with that of individual models like Roberta, CNN, and LSTM. The new hybrid model exhibits a more compact architecture while maintaining a relatively high accuracy rate, boasting an impressive 91%. This hybrid architecture can provide a good compromise between efficiency and performance. This research contributes to the ongoing exploration of efficient and accurate models for textual emotion recognition, emphasizing the significance of balancing model complexity and training speed in practical applications.

Impact of emotional states on tinnitus sound therapy efficacy based on ECG signals and emotion recognition model

BackgroundDiagnosis and severity assessment of tinnitus are mostly based on the patient's descriptions and subjective questionnaires, which lacks objective means of diagnosis and assessment bases, the accuracy of which fluctuates with the clarity of the patient's description. This complicates the timely modification of treatment strategies or therapeutic music to improve treatment efficacy. New methodWe employed a novel random convolutional kernel-based method for electrocardiogram (ECG) signal analysis to identify patients' emotional states during Music Tinnitus Sound Therapy (Music-TST) sessions. Then analyzed correlations between emotional changes in different treatment phase and Tinnitus Handicap Inventory (THI) score differences to determine the impact of emotions on tinnitus treatment efficacy. ResultsThis study revealed a significant correlation between patients' emotion changes during Music-TST and the therapy's effectiveness. Changes in arousal and dominance dimension, were strongly linked to THI variations. These findings highlight the substantial impact of emotional responses on sound therapy's efficacy, offering a new perspective for understanding and optimizing tinnitus treatment. Comparison with existing methodsCompared to existing methods, we proposed an objective indicator to assess the progress of sound therapy, the indicator could also be used to provide feedback to optimize sound therapy music. ConclusionsThis study revealed the critical role of emotion changes in tinnitus sound therapy. By integrating objective ECG-based emotion analysis with traditional subjective scale like THI, we present an innovative approach to assess and potentially optimize therapy effectiveness. This finding could lead to more personalized and effective treatment strategies for tinnitus sound therapy.

Maximizing Emotion Recognition Accuracy with Ensemble Techniques on EEG Signals

Background:: Emotion is a strong feeling such as love, anger, fear, etc. Emotion can be recognized in two ways, i.e., External expression and Biomedical data-based. Nowadays, various research is occurring on emotion classification with biomedical data. Aim:: One of the most current studies in the medical sector, gaming-based applications, education sector, and many other domains is EEG-based emotion identification. The existing research on emotion recognition was published using models like KNN, RF Ensemble, SVM, CNN, and LSTM on biomedical EEG data. In general, only a few works have been published on ensemble or concatenation models for emotion recognition on EEG data and achieved better results than individual ones or a few machine learning approaches. Various papers have observed that CNN works better than other approaches for extracting features from the dataset, and LSTM works better on the sequence data. Method:: Our research is based on emotion recognition using EEG data, a mixed-model deep learning methodology, and its comparison with a machine learning mixed-model methodology. In this study, we introduced a mixed model using CNN and LSTM that classifies emotions in valence and arousal on the DEAP dataset with 14 channels across 32 people. Result and Discussion: We then compared it to SVM, KNN, and RF Ensemble, and concatenated these models with it. First preprocessed the raw data, then checked emotion classification using SVM, KNN, RF Ensemble, CNN, and LSTM individually. After that with the mixed model of CNN-LSTM, and SVM-KNN-RF Ensemble results are compared. Proposed model results have better accuracy as 80.70% in valence than individual ones with CNN, LSTM, SVM, KNN, RF Ensemble and concatenated models of SVM, KNN and RF Ensemble. Conclusion:: Overall, this paper concludes a powerful technique for processing a range of EEG data is the combination of CNNs and LSTMs. Ensemble approach results show better performance in the case of valence at 80.70% and 78.24% for arousal compared to previous research.

CCNN-SVM: Automated Model for Emotion Recognition Based on Custom Convolutional Neural Networks with SVM

The expressions on human faces reveal the emotions we are experiencing internally. Emotion recognition based on facial expression is one of the subfields of social signal processing. It has several applications in different areas, specifically in the interaction between humans and computers. This study presents a simple CCNN-SVM automated model as a viable approach for FER. The model combines a Convolutional Neural Network for feature extraction, certain image preprocessing techniques, and Support Vector Machine (SVM) for classification. Firstly, the input image is preprocessed using face detection, histogram equalization, gamma correction, and resizing techniques. Secondly, the images go through custom single Deep Convolutional Neural Networks (CCNN) to extract deep features. Finally, SVM uses the generated features to perform the classification. The suggested model was trained and tested on four datasets, CK+, JAFFE, KDEF, and FER. These datasets consist of seven primary emotional categories, which encompass anger, disgust, fear, happiness, sadness, surprise, and neutrality for CK+, and include contempt for JAFFE. The model put forward demonstrates commendable performance in comparison to existing facial expression recognition techniques. It achieves an impressive accuracy of 99.3% on the CK+ dataset, 98.4% on the JAFFE dataset, 87.18% on the KDEF dataset, and 88.7% on the FER.

Evolutionary Ensemble Learning for EEG-Based Cross-Subject Emotion Recognition.

Electroencephalogram (EEG) has been widely utilized in emotion recognition due to its high temporal resolution and reliability. However, the individual differences and non-stationary characteristics of EEG, along with the complexity and variability of emotions, pose challenges in generalizing emotion recognition models across subjects. In this paper, an end-to-end framework is proposed to improve the performance of cross-subject emotion recognition. A novel evolutionary programming (EP)-based optimization strategy with neural network (NN) as the base classifier termed NN ensemble with EP (EPNNE) is designed for cross-subject emotion recognition. The effectiveness of the proposed method is evaluated on the publicly available DEAP, FACED, SEED, and SEED-IV datasets. Numerical results demonstrate that the proposed method is superior to state-of-the-art cross-subject emotion recognition methods. The proposed end-to-end framework for cross-subject emotion recognition aids biomedical researchers in effectively assessing individual emotional states, thereby enabling efficient treatment and interventions.

Real-Time Classroom Behavior Analysis for Enhanced Engineering Education: An AI-Assisted Approach

Modern teaching has made significant progress, with many advanced equipment and technologies being introduced into the teaching process. Experimental teaching of engineering design courses is important. Due to limited teaching resources, engineering students need effective guidance during limited laboratory time. We will introduce artificial intelligence solutions to engineering education. We will use artificial intelligence technology for classroom behavior analysis to improve engineering design practice courses' teaching effectiveness. In an instructional milieu, image acquisition tools such as cameras are capable of real-time data capture, facilitating the identification and enumeration of students' emotional states. Concurrently, analytical software gauges the students' interaction patterns and performs comprehensive cluster analysis. Such multifaceted information provides valuable insights into the students' educational engagement, allowing educators to tailor their approach, thereby fostering enhanced pedagogical outcomes. The emotion recognition model we have developed, namely ERAM, demonstrates a rapid response rate coupled with dependable accuracy, making it a robust tool for classroom implementation. In contrast to the conventional post-lesson evaluations, our proposed technique furnishes immediate feedback throughout the instructional process. This real-time approach heralds a significant shift in instructional methodology, promoting timely intervention and adaptive teaching strategies. The control group experiment showed that intelligent systems improved teaching effectiveness by 8.44%. Intelligent systems can help teachers understand students' learning status and improve laboratory teaching quality in engineering design courses.

Emotion Recognition in Lhasa Tibetan Speech based on Bi-LSTM Graph Convolutional Networks

Speech Emotion Recognition (SER) is a crucial component in the field of Human-Computer Interaction (HCI), with significant research and practical application implications. However, due to the complexity of the Tibetan language and the scarcity of datasets caused by the difficulty in collecting various dialects, there are not many research achievements in Tibetan speech recognition. Based on the foundation of constructing a TBLS1 dataset containing 6,000 Tibetan-language speech samples, an approach was devised for Tibetan speech emotion recognition. This approach leverages MFCC features and incorporates a Bi-directional Long Short-Term Memory (Bi-LSTM) network within a graph convolutional neural network. Finally, by comparing the performance of different models on this dataset, we demonstrated the feasibility of our model for Tibetan speech emotion recognition.

EEG Emotion Recognition Employing RGPCN-BiGRUAM: ReliefF-Based Graph Pooling Convolutional Network and BiGRU Attention Mechanism

Emotion recognition plays a crucial role in affective computing, and electroencephalography (EEG) signals are increasingly applied in this field due to their effectiveness in reflecting brain activity. In this paper, we propose a novel EEG emotion recognition model that combines the ReliefF-based Graph Pooling Convolutional Network and BiGRU Attention Mechanisms (RGPCN-BiGRUAM). RGPCN-BiGRUAM effectively integrates the advantages of graph convolutional networks and recurrent neural networks. By incorporating ReliefF weights and an attention mechanism into graph pooling, our model enhances the aggregation of high-quality features while discarding irrelevant ones, thereby improving the efficiency of information transmission. The implementation of a multi-head attention mechanism fusion in BiGRU addresses the limitations of single-output features, achieving optimal selection of global features. Comparative experiments on public datasets SEED and DEAP demonstrate that our proposed RGPCN-BiGRUAM significantly improves classification performance compared to classic algorithms, achieving state-of-the-art results. Ablation studies further validate the design principles of our model. The results of this study indicate that RGPCN-BiGRUAM has strong potential for EEG emotion recognition, offering substantial possibilities for future applications.

Speech Emotion Recognition Incorporating Relative Difficulty and Labeling Reliability.

Emotions in speech are expressed in various ways, and the speech emotion recognition (SER) model may perform poorly on unseen corpora that contain different emotional factors from those expressed in training databases. To construct an SER model robust to unseen corpora, regularization approaches or metric losses have been studied. In this paper, we propose an SER method that incorporates relative difficulty and labeling reliability of each training sample. Inspired by the Proxy-Anchor loss, we propose a novel loss function which gives higher gradients to the samples for which the emotion labels are more difficult to estimate among those in the given minibatch. Since the annotators may label the emotion based on the emotional expression which resides in the conversational context or other modality but is not apparent in the given speech utterance, some of the emotional labels may not be reliable and these unreliable labels may affect the proposed loss function more severely. In this regard, we propose to apply label smoothing for the samples misclassified by a pre-trained SER model. Experimental results showed that the performance of the SER on unseen corpora was improved by adopting the proposed loss function with label smoothing on the misclassified data.

Deep learning-based dimensional emotion recognition for conversational agent-based cognitive behavioral therapy.

Internet-based cognitive behavioral therapy (iCBT) offers a scalable, cost-effective, accessible, and low-threshold form of psychotherapy. Recent advancements explored the use of conversational agents such as chatbots and voice assistants to enhance the delivery of iCBT. These agents can deliver iCBT-based exercises, recognize and track emotional states, assess therapy progress, convey empathy, and potentially predict long-term therapy outcome. However, existing systems predominantly utilize categorical approaches for emotional modeling, which can oversimplify the complexity of human emotional states. To address this, we developed a transformer-based model for dimensional text-based emotion recognition, fine-tuned with a novel, comprehensive dimensional emotion dataset comprising 75,503 samples. This model significantly outperforms existing state-of-the-art models in detecting the dimensions of valence, arousal, and dominance, achieving a Pearson correlation coefficient of r=0.90, r=0.77, and r=0.64, respectively. Furthermore, a feasibility study involving 20 participants confirmed the model's technical effectiveness and its usability, acceptance, and empathic understanding in a conversational agent-based iCBT setting, marking a substantial improvement in personalized and effective therapy experiences.

Exploring emotional intelligence in artificial intelligence systems: a comprehensive analysis of emotion recognition and response mechanisms.

This study aims to dissect the current state of emotion recognition and response mechanisms in artificial intelligence (AI) systems, exploring the progress made, challenges faced, and implicit operations of integrating emotional intelligence into AI. This study utilized a comprehensive review approach to investigate the integration of emotional intelligence (EI) into artificial intelligence (AI) systems, concentrating on emotion recognition and response mechanisms. The review process entailed formulating research questions, systematically searching academic databases such as PubMed, Scopus, and Web of Science, critically evaluating relevant literature, synthesizing the data, and presenting the findings in a comprehensive format. The study highlights the advancements in emotion recognition models, including the use of deep literacy ways and multimodal data emulsion. It discusses the challenges in emotion recognition, similar to variability in mortal expressions and the need for real-time processing. The integration of contextual information and individual traits is emphasized as enhancing the understanding of mortal feelings. The study also addresses ethical enterprises, similar as sequestration and impulses in training data. The integration of emotional intelligence into AI systems presents openings to revise mortal-computer relations. Emotion recognition and response mechanisms have made significant progress, but challenges remain. Unborn exploration directions include enhancing the robustness and interpretability of emotion recognition models, exploring cross-cultural and environment-apprehensive emotion understanding, and addressing long-term emotion shadowing and adaption. By further exploring emotional intelligence in AI systems, further compassionate and responsive machines can be developed, enabling deeper emotional connections with humans.

Integrating artificial intelligence to assess emotions in learning environments: a systematic literature review.

Artificial Intelligence (AI) is transforming multiple sectors within our society, including education. In this context, emotions play a fundamental role in the teaching-learning process given that they influence academic performance, motivation, information retention, and student well-being. Thus, the integration of AI in emotional assessment within educational environments offers several advantages that can transform how we understand and address the socio-emotional development of students. However, there remains a lack of comprehensive approach that systematizes advancements, challenges, and opportunities in this field. This systematic literature review aims to explore how artificial intelligence (AI) is used to evaluate emotions within educational settings. We provide a comprehensive overview of the current state of research, focusing on advancements, challenges, and opportunities in the domain of AI-driven emotional assessment within educational settings. The review involved a search across the following academic databases: Pubmed, Web of Science, PsycINFO and Scopus. Forty-one articles were selected that meet the established inclusion criteria. These articles were analyzed to extract key insights related to the integration of AI and emotional assessment within educational environments. The findings reveal a variety of AI-driven approaches that were developed to capture and analyze students' emotional states during learning activities. The findings are summarized in four fundamental topics: (1) emotion recognition in education, (2) technology integration and learning outcomes, (3) special education and assistive technology, (4) affective computing. Among the key AI techniques employed are machine learning and facial recognition, which are used to assess emotions. These approaches demonstrate promising potential in enhancing pedagogical strategies and creating adaptive learning environments that cater to individual emotional needs. The review identified emerging factors that, while important, require further investigation to understand their relationships and implications fully. These elements could significantly enhance the use of AI in assessing emotions within educational settings. Specifically, we are referring to: (1) federated learning, (2) convolutional neural network (CNN), (3) recurrent neural network (RNN), (4) facial expression databases, and (5) ethics in the development of intelligent systems. This systematic literature review showcases the significance of AI in revolutionizing educational practices through emotion assessment. While advancements are evident, challenges related to accuracy, privacy, and cross-cultural validity were also identified. The synthesis of existing research highlights the need for further research into refining AI models for emotion recognition and emphasizes the importance of ethical considerations in implementing AI technologies within educational contexts.

FCAN : Speech emotion recognition network based on focused contrastive learning

Traditional speech emotion recognition (SER) models predominantly utilize cross-entropy loss for supervised optimization. Yet, given the limited size of SER datasets, relying solely on cross-entropy loss does not adequately capture the intricacies within the data. Moreover, conventional contrastive learning techniques, though improving performance, do not completely harness the inherent data structure across samples. To surmount these limitations, we introduce an innovative emotion recognition network founded on focused contrastive learning. Our model employs bidirectional gated recurrent unit (GRU) networks to seize critical contextual dependencies. By integrating an attention mechanism, we adeptly merge the contextual features of dialogues with individual utterance attributes, forming a comprehensive multi-stage feature fusion framework. This strategy extracts more elaborate emotional features from the constrained dataset. Additionally, by amalgamating the focal loss function into our training protocol, we mitigate imbalanced sample issues to a degree. We further refine our contrastive loss function by enacting a weighted strategy for intra-dialogue category assignments, where higher penalty weights are ascribed to harder-to-identify categories, thus incentivizing the model to concentrate on distinguishing subtler category differences. This refinement significantly bolsters the model’s proficiency in learning various emotional feature distributions, thereby enhancing its emotion recognition capabilities. Experimental validation on IEMOCAP and MELD benchmark datasets underpins our model’s superiority.

Enhanced Emotion Detection Model Using Dusky Canidae Optimization-Based Deep CNN Classifier

Emotion detection involves distinguishing the complex nature of emotions, cross-cultural variations, and ethical considerations. The proposed model addresses cross-cultural issues through diverse training and ensures ethical data usage through advanced machine-learning techniques to enhance accuracy. The research aims to develop a Dusky Canidae Optimization-based Deep Convolutional Neural Network model (DCO opt deep CNN) for emotion recognition in audio-visual data, benefiting various advancements. During preprocessing, audio and video are separated to enhance sound quality and eliminate data redundancy while retaining vital information. Features are then extracted and employed for training and testing using the automation and effectiveness of Deep CNNs. The concatenated features are optimized using a hybrid approach that combines Dusky Hawk and Canidae Wolf Optimization (CWO), offering simplicity and interpretability with optimal parameter settings. The proposed DCO model effectively classifies emotions within the data. Evaluating the database, the DCO model exhibits a performance enhancement of 1.28%, 2.91%, and 1.28% in accuracy, sensitivity, and specificity with respect to the existing learner memorizing optimization-based deep NN model.

Speech Emotion Recognition under Noisy Environments with SNR Down to −6 dB Using Multi-Decoder Wave-U-Net

A speech emotion recognition (SER) model for noisy environments is proposed, by using four band-pass filtered speech waveforms as the model input instead of the simplified input features such as MFCC (Mel Frequency Cepstral Coefficients). The four waveforms retain the entire information of the original noisy speech while the simplified features keep only partial information of the noisy speech. The information reduction at the model input may cause the accuracy degradation under noisy environments. A normalized loss function is used for training to maintain the high-frequency details of the original noisy speech waveform. A multi-decoder Wave-U-Net model is used to perform the denoising operation and the Wave-U-Net output waveform is applied to an emotion classifier in this work. By this, the number of parameters is reduced to 2.8 M for inference from 4.2 M used for training. The Wave-U-Net model consists of an encoder, a 2-layer LSTM, six decoders, and skip-nets; out of the six decoders, four decoders are used for denoising four band-pass filtered waveforms, one decoder is used for denoising the pitch-related waveform, and one decoder is used to generate the emotion classifier input waveform. This work gives much less accuracy degradation than other SER works under noisy environments; compared to accuracy for the clean speech waveform, the accuracy degradation is 3.8% at 0 dB SNR in this work while it is larger than 15% in the other SER works. The accuracy degradation of this work at SNRs of 0 dB, −3 dB, and −6 dB is 3.8%, 5.2%, and 7.2%, respectively.

Audio-visual expression-based emotion recognition model for neglected people in real-time: a late-fusion approach

Audio-visual expression-based emotion recognition model for neglected people in real-time: a late-fusion approach

Exploration of a method for insight into accessibility design flaws based on touch dynamics

The widespread use of smartphones implies that intelligent systems must address the negative emotions experienced by a diverse user group when encountering usability difficulties. Therefore, exploring the relationship between users’ emotions and operational behaviors is crucial for further enhancing the accessibility of intelligent systems. Touch dynamics, as a method of dynamic touch data monitoring, has already provided many models for emotion recognition, which are beneficial in gaining insight into users’ accessibility needs for smartphones and exploring intelligent solutions to design issues. This paper validates this aspect through interviews and experimental studies. The study first conducted an open-ended interview, and then used a touch behavior experiment with emotional labels and a questionnaire to validate the association between accessibility design flaws in intelligent systems and user emotions. The interview results show that middle-aged and older smartphone users have difficulty activating accessibility tools independently. Consequently, this paper establishes a dataset for emotion classification based on touch gesture data, which is used to gain insight into implicit difficulties in the interaction process through negative emotions. From a touch experiment and questionnaire survey of 57 sample users, there is approximately a 78.57% match between the accessibility needs of intelligent systems and users’ negative emotions, with interaction fluency and interface cognition being the main reasons for this phenomenon. Therefore, touch dynamics, as a non-invasive method of emotion recognition, can use negative emotional feedback to gain insights into user needs that are difficult to detect by other methods. This has application value for enhancing users’ emotional experience and promoting a transformation in interaction design thinking.

Enhancing speech emotion recognition with deep learning using multi-feature stacking and data augmentation

This study evaluates the effectiveness of data augmentation on 1D convolutional neural network (CNN) and transformer models for speech emotion recognition (SER) on the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset. The results show that data augmentation has a positive impact on improving emotion classification accuracy. Techniques such as noising, pitching, stretching, shifting, and speeding are applied to increase data variation and overcome class imbalance. The 1D CNN model with data augmentation achieved 94.5% accuracy, while the transformer model with data augmentation performed even better at 97.5%. This research is expected to contribute better insights for the development of accurate emotion recognition methods by using data augmentation with these models to improve classification accuracy on the RAVDESS dataset. Further research can explore larger and more diverse datasets and alternative model approaches.

A high speed inference architecture for multimodal emotion recognition based on sparse cross modal encoder

In recent years, multimodal emotion recognition models are using pre-trained networks and attention mechanisms to pursue higher accuracy, which increases the training burden and slows down the training and inference speed. In order to strike a balance between speed and accuracy, this paper proposes a speed-optimized multimodal emotion recognition architecture for speech and text emotion recognition. In the feature extraction part, a lightweight residual graph convolutional network (ResGCN) is selected as the speech feature extractor, and an efficient RoBERTa pre-trained network is used as the text feature extractor. Then, an algorithm complexity-optimized sparse cross-modal encoder (SCME) is proposed and used to fuse these two types of features. Finally, a new gated fusion module (GF) is used to weight multiple results and input them into a fully connected layer (FC) for classification. The proposed method is tested on the IEMOCAP dataset and the MELD dataset, achieving weighted accuracies (WA) of 82.4% and 65.0%, respectively. This method achieves higher accuracy than the listed methods while having an acceptable training and inference speed.