Emotion Recognition System Research Articles

TheraSense: Deep Learning for Facial Emotion Analysis in Mental Health Teleconsultation

Background: This paper presents TheraSense, a system developed within the Supporting Mental Health in Young People: Integrated Methodology for cLinical dEcisions and evidence (Smile) and Sensor Enabled Affective Computing for Enhancing Medical Care (SenseCare) projects. TheraSense is designed to enhance teleconsultation services by leveraging deep learning for real-time emotion recognition through facial expressions. It integrates with the Knowledge Management-Ecosystem Portal (SenseCare KM-EP) platform to provide mental health practitioners with valuable emotional insights during remote consultations. Method: We describe the conceptual design of TheraSense, including its use case contexts, architectural structure, and user interface layout. The system’s interoperability is discussed in detail, highlighting its seamless integration within the teleconsultation workflow. The evaluation methods include both quantitative assessments of the video-based emotion recognition system’s performance and qualitative feedback through heuristic evaluation and survey analysis. Results: The performance evaluation shows that TheraSense effectively recognizes emotions in video streams, with positive user feedback on its usability and integration. The system’s real-time emotion detection capabilities provide valuable support for mental health practitioners during remote sessions. Conclusions: TheraSense demonstrates its potential as an innovative tool for enhancing teleconsultation services. By providing real-time emotional insights, it supports better-informed decision-making in mental health care, making it an effective addition to remote telehealth platforms.

Multimodal emotion recognition system for e-learning platform

Multimodal emotion recognition system for e-learning platform

Advancing Emotion Recognition in Wearable Devices: Core Technologies, Challenges and Practical Solutions

This paper reviews the advancements and challenges in emotion recognition systems, focusing on their applications in wearable devices. It examines core technologiesmultimodal data fusion, deep learning models, and personalized frameworksin addressing critical issues such as data quality, computational efficiency, and privacy protection. By highlighting innovative solutions, including edge computing, lightweight architectures, and advanced privacy-preserving techniques, the paper emphasizes their role in enhancing the robustness, scalability, and security. By integrating real-world applications with state-of-the-art methodologies, this review underscores the transformative potential of wearable emotion recognition technologies for enhancing well-being and daily life. This review also examines current challenges in emotion recognition and provides potential solutions to address these issues

Emotion Recognition from EEG Signals Using Advanced Transformations and Deep Learning

Affective computing aims to develop systems capable of effectively interacting with people through emotion recognition. Neuroscience and psychology have established models that classify universal human emotions, providing a foundational framework for developing emotion recognition systems. Brain activity related to emotional states can be captured through electroencephalography (EEG), enabling the creation of models that classify emotions even in uncontrolled environments. In this study, we propose an emotion recognition model based on EEG signals using deep learning techniques on a proprietary database. To improve the separability of emotions, we explored various data transformation techniques, including Fourier Neural Networks and quantum rotations. The convolutional neural network model, combined with quantum rotations, achieved a 95% accuracy in emotion classification, particularly in distinguishing sad emotions. The integration of these transformations can further enhance overall emotion recognition performance.

Advancing Emotion Recognition: EEG Analysis and Machine Learning for Biomedical Human–Machine Interaction

Background: Human emotions are subjective psychophysiological processes that play an important role in the daily interactions of human life. Emotions often do not manifest themselves in isolation; people can experience a mixture of them and may not express them in a visible or perceptible way; Methods: This study seeks to uncover EEG patterns linked to emotions, as well as to examine brain activity across emotional states and optimise machine learning techniques for accurate emotion classification. For these purposes, the DEAP dataset was used to comprehensively analyse electroencephalogram (EEG) data and understand how emotional patterns can be observed. Machine learning algorithms, such as SVM, MLP, and RF, were implemented to predict valence and arousal classifications for different combinations of frequency bands and brain regions; Results: The analysis reaffirms the value of EEG as a tool for objective emotion detection, demonstrating its potential in both clinical and technological contexts. By highlighting the benefits of using fewer electrodes, this study emphasises the feasibility of creating more accessible and user-friendly emotion recognition systems; Conclusions: Further improvements in feature extraction and model generalisation are necessary for clinical applications. This study highlights not only the potential of emotion classification to develop biomedical applications, but also to enhance human–machine interaction systems.

The Application and Analysis of Emotion Recognition Based on Modern Technology

This article provides a comprehensive analysis of various emotion recognition methods, focusing on speech emotion recognition, facial expression recognition, and physiological signal emotion recognition. The primary aim is to evaluate the advantages and disadvantages of these methods, offering insights into selecting the most appropriate approach for different application scenarios. The study involves collecting and analysing experimental data, exploring their respective strengths and limitations, and proposing potential solutions to enhance their effectiveness. Speech emotion recognition is effective but sensitive to noise and speaker variability, while facial expression recognition excels under controlled conditions but struggles with changes in lighting and angles. Physiological signal recognition offers deep insights into internal emotional states but requires complex signal processing and is vulnerable to external interferences. Despite the growing application of emotion recognition technology across various fields, including healthcare, traffic safety, and security, there remain significant challenges related to accuracy, robustness, and privacy. This study highlights the need for continued research to improve these technologies, particularly in enhancing their robustness and adaptability. The findings provide valuable guidance for researchers and practitioners seeking to optimize emotion recognition systems for diverse real-world applications.

Development of speech emotion recognition system using optimized convolutional neural network

Development of speech emotion recognition system using optimized convolutional neural network

Emotion Recognition from Spontaneous Tunisian Dialect Speech

Emotional expressions are a fundamental aspect of human communication, with speech being one of the most natural modes of interaction. Speech Emotion Recognition (SER) is a significant research topic in Natural Language Processing (NLP), aimed at identifying emotions such as satisfaction, frustration, and anger from speech audio using multiple classifiers. This paper presents a method to emotion recognition from spontaneous Tunisian Dialect (TD) speech, marking the first work in the SER field to utilize spontaneous speech for emotion recognition in this dialect. The dataset was created from freely available YouTube videos across multiple domains and labeled with four perceived emotions: anger, satisfaction, frustration, and neutral. To address the data scarcity issue, we implemented data augmentation techniques, specifically Vocal Tract Length Perturbation (VTLP). The preprocessing of the speech signals involved cleaning the data from ambient and unwanted noises. We extracted and selected various spectral features, including mel-frequency cepstral coefficients (MFCC) and Linear Prediction Cepstral Coefficients (LPCC). Subsequently, we applied several classification methods: Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Random Forest. Our experiments demonstrated that the Random Forest classifier achieved the highest F-score of 58.75%. The results were thoroughly discussed, analyzed, and compared across the five models using different feature extractions. This study provides valuable insights and advancements in the SER field, particularly for the TD, future research directions for improving emotion recognition systems.

A Survey of Multi-modal Emotion Recognition Based on Deep Learning

Multi-modal emotion recognition technology explores emotion recognition by integrating facial expression, voice intonation, text analysis and other multi-source data, so as to improve the naturalness and accuracy of human-computer interaction. Aiming at the emerging field of multi-modal emotion recognition, this paper introduces three single-modal emotion recognition methods of text, face and voice, especially the problem of multi-modal emotion fusion, and introduces the methods with high success rate of multi-modal emotion fusion recognition in recent years. Through comparative analysis, the conclusion is drawn that the current fusion methods are more complicated and the fusion success rate has been improved to some extent. However, the number of data sets on multi-modal emotion analysis is small, and the research on gesture and other modes of emotion recognition is also scarce. In the later stage, it is necessary to enrich the data set and add new modes to improve the accuracy and robustness of the multi-modal emotion recognition and analysis system.

A client-server based recognition system: Non-contact single/multiple emotional and behavioral state assessment methods.

In the current global health landscape, there is an increasing demand for rapid and accurate assessment of mental states. Traditional assessment methods typically rely on face-to-face interactions, which are not only time-consuming but also highly subjective. Addressing this issue, this study aims to develop a client-server-based, non-contact multimodal emotion and behavior recognition system to enhance the efficiency and accuracy of mental state assessments. This study designed and implemented a multimodal assessment system integrating voice, text, facial expressions, and body movements. Utilizing a client-server architecture, the system optimizes diagnostic efficiency and decision-making accuracy through an intuitive visual interface. The system's effectiveness was validated and tested in actual hospital settings. The system demonstrated exceptional performance in multimodal emotion and behavior recognition, achieving a voice recognition accuracy of 92.01%, facial expression recognition accuracy of 91.3%, and an overall multimodal assessment accuracy of 77.9%. Moreover, it reached a behavior analysis accuracy of 94.5%. The multimodal assessment system developed in this study significantly enhances the accuracy and efficiency of mental state assessments, meeting the needs of clinicians for precise and rapid diagnostics in real-world settings.

Automated Human Emotion Recognition System Using TQWT-Based EEG Subbands

Automated Human Emotion Recognition System Using TQWT-Based EEG Subbands

Multi-physiological signal fusion for objective emotion recognition in educational human-computer interaction.

An increasing prevalence of psychological stress and emotional issues among higher education teachers necessitates innovative approaches to promote their wellbeing. Emotion recognition technology, integrated into educational human-computer interaction (HCI) systems, offers a promising solution. This study aimed to develop a robust emotion recognition system to enhance teacher-student interactions within educational HCI settings. A multi-physiological signal-based emotion recognition system was developed using wearable devices to capture electrocardiography (ECG), electromyography (EMG), electrodermal activity, and respiratory signals. Feature extraction was performed using time-domain and time-frequency domain analysis methods, followed by feature selection to eliminate redundant features. A convolutional neural network (CNN) with attention mechanisms was employed as the decision-making model. The proposed system demonstrated superior accuracy in recognizing emotional states than existing methods. The attention mechanisms provided interpretability by highlighting the most informative physiological features for emotion classification. The developed system offers significant advancements in emotion recognition for educational HCI, enabling more accurate and standardized assessments of teacher emotional states. Real-time integration of this technology into educational environments can enhance teacher-student interactions and contribute to improved learning outcomes. Future research can explore the generalizability of this system to diverse populations and educational settings.

How Much Does the Dynamic F0 Curve Affect the Expression of Emotion in Utterances?

The modulation of vocal elements, such as pitch, loudness, and duration, plays a crucial role in conveying both linguistic information and the speaker’s emotional state. While acoustic features like fundamental frequency (F0) variability have been widely studied in emotional speech analysis, accurately classifying emotion remains challenging due to the complex and dynamic nature of vocal expressions. Traditional analytical methods often oversimplify these dynamics, potentially overlooking intricate patterns indicative of specific emotions. This study examines the influences of emotion and temporal variation on dynamic F0 contours in the analytical framework, utilizing a dataset valuable for its diverse emotional expressions. However, the analysis is constrained by the limited variety of sentences employed, which may affect the generalizability of the findings to broader linguistic contexts. We utilized the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), focusing on eight distinct emotional states performed by 24 professional actors. Sonorant segments were extracted, and F0 measurements were converted into semitones relative to a 100 Hz baseline to standardize pitch variations. By employing Generalized Additive Mixed Models (GAMMs), we modeled non-linear trajectories of F0 contours over time, accounting for fixed effects (emotions) and random effects (individual speaker variability). Our analysis revealed that incorporating emotion-specific, non-linear time effects and individual speaker differences significantly improved the model’s explanatory power, ultimately explaining up to 66.5% of the variance in the F0. The inclusion of random smooths for time within speakers captured individual temporal modulation patterns, providing a more accurate representation of emotional speech dynamics. The results demonstrate that dynamic modeling of F0 contours using GAMMs enhances the accuracy of emotion classification in speech. This approach captures the nuanced pitch patterns associated with different emotions and accounts for individual variability among speakers. The findings contribute to a deeper understanding of the vocal expression of emotions and offer valuable insights for advancing speech emotion recognition systems.

In-depth Study and Application Analysis of Multimodal Emotion Recognition Methods: Multidimensional Fusion Techniques Based on Vision, Speech, and Text

Abstract. Emotion recognition technology, pivotal in fields such as medical health, game entertainment, and human-computer interaction, benefits significantly from multimodal approaches. This paper delves into the techniques and applications of multimodal emotion recognition, focusing on fusion methods that integrate visual, speech, and text data. Emotion recognition through single modalities often faces limitations such as susceptibility to noise and low accuracy, whereas multimodal systems exhibit enhanced performance by leveraging combined data sources. The primary fusion techniques discussed include feature-level, decision-level, and model-level integrations. Feature-level fusion amalgamates multiple data types early in the processing stage, improving detection robustness. Decision-level fusion, on the other hand, involves synthesizing results from separate analyses, offering flexibility and ease of integration. Model-level fusion allows for deep interactions between modalities, potentially capturing more complex emotional states. This study confirms that multimodal emotion recognition systems generally surpass the performance of their single-modal counterparts, advocating for further exploration into sophisticated fusion techniques to boost accuracy and applicability across various domains.

Real-Time Facial Emotion Detection Application with Image Processing Based on Convolutional Neural Network (CNN)

Facial Emotion Recognition (FER) is a key technology for identifying emotions based on facial expressions, with applications in human-computer interaction, mental health monitoring, and customer analysis. This study presents the development of a real-time emotion recognition system using Convolutional Neural Networks (CNNs) and OpenCV, addressing challenges such as varying lighting and facial occlusions. The system, trained on the FER2013 dataset, achieved 85% accuracy in emotion classification, demonstrating high performance in detecting happiness, sadness, and surprise. The results highlight the system's effectiveness in real-time applications, offering potential for use in mental health and customer behavior analysis.

Emotional AI and Data Protection: Relevant Implications for Older Adults and Other Vulnerable Subjects

By examining the legal framework that has recently emerged in Europe, the work focuses on the main problems caused by the proliferation of AI technologies used for the care and well-being of vulnerable subjects (especially older people) and aims to identify tools of protection that can be used by the subject-user in the event of discrimination and violations of fundamental rights. One of the areas which has recently been most affected by AI is that concerning the use of emotion recognition systems for the health-care and well-being of these individuals, who are not offered secure protection for self-determination and privacy. In light of an analysis of the rapid advancement of these systems, the article also examines, through a comparative perspective, the regulatory challenges and the issues faced by Europe and the US.

Polish Speech and Text Emotion Recognition in a Multimodal Emotion Analysis System

Emotion recognition by social robots is a serious challenge because sometimes people also do not cope with it. It is important to use information about emotions from all possible sources: facial expression, speech, or reactions occurring in the body. Therefore, a multimodal emotion recognition system was introduced, which includes the indicated sources of information and deep learning algorithms for emotion recognition. An important part of this system includes the speech analysis module, which was decided to be divided into two tracks: speech and text. An additional condition is the target language of communication, Polish, for which the number of datasets and methods is very limited. The work shows that emotion recognition using a single source—text or speech—can lead to low accuracy of the recognized emotion. It was therefore decided to compare English and Polish datasets and the latest deep learning methods in speech emotion recognition using Mel spectrograms. The most accurate LSTM models were evaluated on the English set and the Polish nEMO set, demonstrating high efficiency of emotion recognition in the case of Polish data. The conducted research is a key element in the development of a decision-making algorithm for several emotion recognition modules in a multimodal system.

Speech Emotion Recognition Algorithm of Intelligent Robot Based on ACO-SVM

Despite the significant advancements in computer speech emotion recognition technology, the deployment of intelligent robots in this domain continues to encounter challenges related to inefficiency and emotional ambiguity. Aiming at the shortcomings of existing algorithms in the accuracy and processing of complex emotion states, a novel emotion recognition model of intelligent robot speech based on ACO algorithm and multi-level support vector machine is proposed. This paper first identifies the efficiency and accuracy problems existing in the current emotion recognition system, and clarified the research gap, that is, how to effectively improve the recognition performance of the model under diverse emotion states. To solve this problem, genetic algorithm is used to fine-tune SVM model parameters, and multi-level SVM classification architecture is constructed to enhance the accuracy of emotion recognition. In the feature selection stage, ACO algorithm is introduced to explore the optimal combination of emotion features, aiming at improving the efficiency and robustness of emotion recognition. The experimental results showed that the average recognition rate of the multilevel SVM model optimized by genetic algorithm was as high as 81.26%, which was 8.78 percentage points higher than that before optimization. When the ACO algorithm was combined with the optimized multi-stage SVM, the average recognition rate of German and Chinese speech emotion corpus was further improved to 85.62%, with an improvement of 4.38 percentage points. In particular, the optimized algorithm performed significantly better than other emotion categories in recognizing complex emotions such as "anger" and "sadness." This paper not only offers an innovative solution for the emotion recognition technology within the realm of intelligent robots, but also extends its methodology to have significant implications for related fields, thereby strongly advancing the in-depth application and development of intelligent bionic algorithms in speech emotion recognition. In addition, through this series of optimization strategies, this paper significantly improves the accuracy of emotion recognition, laying a solid foundation for intelligent robots to achieve more natural and efficient human-computer interaction.

Emotion Recognition in Kurdish Speech from the Sorani Dialect Corpus

Given the increasing need for interactive human-computer applications, the field of employing machine learning algorithms to discern emotions from speech has seen a substantial surge in interest. While emotion recognition systems have made substantial progress in languages like German, English, Spanish, Dutch, and Danish, the availability of comprehensive datasets for the Kurdish language remains notably limited. This paper addresses this gap by focusing on emotion recognition in Sorani Kurdish dialect speech data, which was carefully gathered from openly available videos from the YouTube platform and categorized into four clear supposed emotions: neutral, sadness, happiness, and anger. The study applied both natural Mel Spectrogram and Mel-Frequency Cepstral Coefficient (MFCC) features for various spectrals, followed by the classification models K-Nearest Neighbor (KNN), Multi-Layer Perceptron (MLP), and Support Vector Machine (SVM) to evaluate the results. By closely examining and contrasting the results of using several methods for feature extraction, it was found that SVM obtained a higher accuracy, reaching as much as 85.57%. This is so much more than the first Kurdish emotion classification technique for the recognition of the emotion of the words.

Integrating Diverse Data Streams for Enhanced Emotional Intelligence in Mental Health Care

This research presents a comprehensive system for real-time emotion recognition and analysis using multimodal data, including image, video, audio, and text. The system employs deep learning models to extract features and classify emotions from each modality. By integrating these predictions, we aim to provide a holistic understanding of a user's emotional state and assess potential risks. The system further analyzes the collected emotion data to identify trends, patterns, and indicators of emotional well-being and suicide risk. Visualizations such as time-series plots, distribution charts, and stacked area charts are employed to represent the emotional dynamics over time.Based on the analysis; the system generates personalized recommendations and alerts for users exhibiting signs of distress or emotional instability, including potential suicide risk factors. The goal is to promote proactive well-being management and early intervention. It is crucial to emphasize that this system is a tool for early detection and should be used in conjunction with professional mental health care.