Real-time Emotion Research Articles

Utilizing virtual reality for real-time emotion recognition with artificial intelligence: a systematic literature review

Utilizing virtual reality for real-time emotion recognition with artificial intelligence: a systematic literature review

TheraSense: Deep Learning for Facial Emotion Analysis in Mental Health Teleconsultation

Background: This paper presents TheraSense, a system developed within the Supporting Mental Health in Young People: Integrated Methodology for cLinical dEcisions and evidence (Smile) and Sensor Enabled Affective Computing for Enhancing Medical Care (SenseCare) projects. TheraSense is designed to enhance teleconsultation services by leveraging deep learning for real-time emotion recognition through facial expressions. It integrates with the Knowledge Management-Ecosystem Portal (SenseCare KM-EP) platform to provide mental health practitioners with valuable emotional insights during remote consultations. Method: We describe the conceptual design of TheraSense, including its use case contexts, architectural structure, and user interface layout. The system’s interoperability is discussed in detail, highlighting its seamless integration within the teleconsultation workflow. The evaluation methods include both quantitative assessments of the video-based emotion recognition system’s performance and qualitative feedback through heuristic evaluation and survey analysis. Results: The performance evaluation shows that TheraSense effectively recognizes emotions in video streams, with positive user feedback on its usability and integration. The system’s real-time emotion detection capabilities provide valuable support for mental health practitioners during remote sessions. Conclusions: TheraSense demonstrates its potential as an innovative tool for enhancing teleconsultation services. By providing real-time emotional insights, it supports better-informed decision-making in mental health care, making it an effective addition to remote telehealth platforms.

Temporal dynamics of affective scene processing in the healthy adult human brain.

Understanding how the brain distinguishes emotional from neutral scenes is crucial for advancing brain-computer interfaces, enabling real-time emotion detection for faster, more effective responses, and improving treatments for emotional disorders like depression and anxiety. However, inconsistent research findings have arisen from differences in study settings, such as variations in the time windows, brain regions, and emotion categories examined across studies. This review sought to compile the existing literature on the timing at which the adult brain differentiates basic affective from neutral scenes in less than one second, as previous studies have consistently shown that the brain can begin recognizing emotions within just a few milliseconds. The review includes studies that used electroencephalography (EEG) or magnetoencephalography (MEG) in healthy adults to examine brain responses to emotional versus neutral images within one second. Articles of interest were limited to the English language but not to any publication year. Excluded studies involved only patients (of any diagnosis), participants under age 18 (since emotional processing can differ between adults and younger individuals), non-passive tasks, low temporal resolution techniques, time intervals over one second, and animals. Of the 3045 screened articles, 19 met these criteria. Despite the variations between studies, the earliest onset for heightened brain responses to basic affective scenes compared to neutral ones was most commonly observed within the 250-300 ms time window. To the best of our knowledge, this review is the first to synthesize data on the timing of brain differentiation between emotional and neutral scenes in healthy adults.

Real-Time Emotion Recognition for Improving the Teaching-Learning Process: A Scoping Review.

Emotion recognition (ER) is gaining popularity in various fields, including education. The benefits of ER in the classroom for educational purposes, such as improving students' academic performance, are gradually becoming known. Thus, real-time ER is proving to be a valuable tool for teachers as well as for students. However, its feasibility in educational settings requires further exploration. This review offers learning experiences based on real-time ER with students to explore their potential in learning and in improving their academic achievement. The purpose is to present evidence of good implementation and suggestions for their successful application. The content analysis finds that most of the practices lead to significant improvements in terms of educational purposes. Nevertheless, the analysis identifies problems that might block the implementation of these practices in the classroom and in education; among the obstacles identified are the absence of privacy of the students and the support needs of the students. We conclude that artificial intelligence (AI) and ER are potential tools to approach the needs in ordinary classrooms, although reliable automatic recognition is still a challenge for researchers to achieve the best ER feature in real time, given the high input data variability.

An Ensemble Kernelized-based Approach for Precise Emotion Recognition in Depressed People

As the COVID-19 pandemic created serious challenges for mental health worldwide, with a noticeable increase in depression cases, it has become important to quickly and accurately assess emotional states. Facial expression recognition technology is a key tool for this task. To address this need, this study proposes a new approach to emotion recognition using the Ensemble Kernelized Learning System (EKLS). Nonverbal cues, such as facial expressions, are crucial in showing emotional states. This study uses the Extended Cohn-Kanade (CK+) dataset, which was enhanced with images and videos from the COVID-19 era related to depression. Each of these images and videos is manually labeled with the corresponding emotions, creating a strong dataset for training and testing the proposed model. Facial feature detection techniques were used along with key facial measurements to aid in emotion recognition. EKLS is a flexible machine-learning framework that combines different techniques, including Support Vector Machines (SVMs), Self-Organizing Maps (SOMs), kernel methods, Random Forest (RF), and Gradient Boosting (GB). The ensemble model was thoroughly trained and fine-tuned to ensure high accuracy and consistency. EKLS is a powerful tool for real-time emotion recognition in both images and videos, achieving an impressive accuracy of 99.82%. This study offers a practical and effective approach to emotion recognition and makes a significant contribution to the field.

LECM: A model leveraging emotion cause to improve real-time emotion recognition in conversations

LECM: A model leveraging emotion cause to improve real-time emotion recognition in conversations

Advanced AI-Driven Emotion Recognition Systems: Applications and Technical Challenges

Focusing on face and landmark analysis, the technical analysis investigates AI-based emotion identification systems' design, application, and difficulties. This thorough article explores the use of these systems in the manufacturing, scientific research, and healthcare industries, emphasizing both creative solutions and technological challenges. The article explores fundamental elements such as real-time emotion analysis frameworks, facial landmark identification systems, convolutional neural networks, and deep learning techniques. It shows how these systems are transforming human-computer interaction while addressing important issues in data protection and ethical implementation through a thorough examination of technical aspects, domain-specific applications, difficulties, and privacy considerations. To shed light on how emotion recognition technology is developing, the study also looks at potential future development paths, such as improvements in scalability, cross-platform integration, and sensor fusion technologies.

Temporal–spatial correlation and graph attention-guided network for micro-expression recognition in English learning livestreams

Micro-expressions, fleeting facial movements lasting 1/25 to 1/3 of a second, offer crucial insights into genuine emotions, particularly valuable in online education settings. The rapid growth of English learning livestreams has heightened the need for accurate, real-time micro-expression recognition to enhance learner engagement and instructional effectiveness. However, existing methods need help with the subtle nature of these expressions, especially in dynamic, low-resolution streaming environments. This paper presents TSG–MER–ELL, a novel end-to-end network for micro-expression recognition in English learning livestreams, integrating temporal–spatial correlation and graph attention mechanisms. The framework addresses the unique challenges of real-time emotion analysis in online language education, where subtle facial cues are crucial in understanding learner engagement and comprehension. The temporal–spatial correlation module employs action units with spatio-temporal graph convolution to aggregate features from diverse facial regions, while transformer encoders construct long-range correlations. The graph attention module builds upon local facial areas to guide self-attention computations, yielding precise local correlation features. These global and local features are fused for the final micro-expression classification. We introduce an adaptive loss function that balances accuracy, efficiency, and relevance to linguistic context. Extensive experiments on SMIC, CASME II, and SAMM datasets, adapted for English learning scenarios, demonstrate TSG–MER–ELL’s superior performance over ten state-of-the-art baselines. The TSG–MER–ELL framework achieves top UF1 and UAR scores across all datasets, significantly improving recognition speed and accuracy. Ablation studies and visualizations of temporal–spatial features and graph attention weights provide insights into the framework’s effectiveness in capturing subtle emotional cues. TSG–MER–ELL’s robust performance in varied online learning conditions highlights its potential to enhance engagement, personalize instruction, and improve overall outcomes in virtual English language education.

Real-Time Facial Emotion Detection Application with Image Processing Based on Convolutional Neural Network (CNN)

Facial Emotion Recognition (FER) is a key technology for identifying emotions based on facial expressions, with applications in human-computer interaction, mental health monitoring, and customer analysis. This study presents the development of a real-time emotion recognition system using Convolutional Neural Networks (CNNs) and OpenCV, addressing challenges such as varying lighting and facial occlusions. The system, trained on the FER2013 dataset, achieved 85% accuracy in emotion classification, demonstrating high performance in detecting happiness, sadness, and surprise. The results highlight the system's effectiveness in real-time applications, offering potential for use in mental health and customer behavior analysis.

LCANet: a model for analysis of students real-time sentiment by integrating attention mechanism and joint loss function

By recognizing students’ facial expressions in actual classroom situations, the students’ emotional states can be quickly uncovered, which can help teachers grasp the students’ learning rate, which allows teachers to adjust their teaching strategies and methods, thus improving the quality and effectiveness of classroom teaching. However, most previous facial expression recognition methods have problems such as missing key facial features and imbalanced class distributions in the dateset, resulting in low recognition accuracy. To address these challenges, this paper proposes LCANet, a model founded on a fused attention mechanism and a joint loss function, which allows the recognition of students’ emotions in real classroom scenarios. The model uses ConvNeXt V2 as the backbone network to optimize the global feature extraction capability of the model, and at the same time, it enables the model to pay closer attention to the key regions in facial expressions. We incorporate an improved Channel Spatial Attention (CSA) module as a way to extract more local feature information. Furthermore, to mitigate the class distribution imbalance problem in the facial expression dataset, we introduce a joint loss function. The experimental results show that our LCANet model has good recognition rates on both the public emotion datasets FERPlus, RAF-DB and AffectNet, with accuracies of 91.43%, 90.03% and 64.43%, respectively, with good robustness and generalizability. Additionally, we conducted experiments using the model in real classroom scenarios, detecting and accurately predicting students’ classroom emotions in real time, which provides an important reference for improving teaching in smart teaching scenarios.

Societal Impact of Machine Learning on Human Emotions

Modern networking conversations generate annotated metadata, necessitating a method for synthesizing insights from statistics. Emotion detection is crucial for practical conversations, distinguishing joy, grief, and wrath. Corpora are becoming the standard for human–machine interaction, aiming to make interactions feel natural and real. A paradigm that identifies debates and customer views can provide a human touch to these interactions. Researchers developed a machine learning framework for assessing emotions in English phrases, utilizing Long Short Term Memory perspective and real-time emotion recognition in idiomatic speech. Emotion recognition rule is created using ontologies like Word Net and Concept Net, Naive Bayes, and Random Forest. Real-time analysis of written words and facial expressions significantly outperforms current algorithms and commandment classifiers in identifying emotional states.

Enhancing Real-Time Emotion Recognition in Classroom Environments Using Convolutional Neural Networks: A Step Towards Optical Neural Networks for Advanced Data Processing

In contemporary academic settings, end-of-semester student feedback on a lecturer’s teaching abilities often fails to provide a comprehensive, real-time evaluation of their proficiency, and becomes less relevant with each new cohort of students. To address these limitations, an innovative feedback method has been proposed, utilizing image processing algorithms to dynamically assess the emotional states of students during lectures by analyzing their facial expressions. This real-time approach enables lecturers to promptly adapt and enhance their teaching techniques. Recognizing and engaging with emotionally positive students has been shown to foster better learning outcomes, as their enthusiasm actively stimulates cognitive engagement and information analysis. The purpose of this work is to identify emotions based on facial expressions using a deep learning model based on a convolutional neural network (CNN), where facial recognition is performed using the Viola–Jones algorithm on a group of students in a learning environment. The algorithm encompasses four key steps: image acquisition, preprocessing, emotion detection, and emotion recognition. The technological advancement of this research lies in the proposal to implement photonic hardware and create an optical neural network which offers unparalleled speed and efficiency in data processing. This approach demonstrates significant advancements over traditional electronic systems in handling computational tasks. An experimental validation was conducted in a classroom with 45 students, demonstrating that the level of understanding in the class as predicted was 43–62.94%, and the proposed CNN algorithm (facial expressions detection) achieved an impressive 83% accuracy in understanding students’ emotional states. The correlation between the CNN deep learning model and the students’ feedback was 91.7%. This novel approach opens avenues for the real-time assessment of students’ engagement levels and the effectiveness of the learning environment, providing valuable insights for ongoing improvements in teaching practices.

Integrating Diverse Data Streams for Enhanced Emotional Intelligence in Mental Health Care

This research presents a comprehensive system for real-time emotion recognition and analysis using multimodal data, including image, video, audio, and text. The system employs deep learning models to extract features and classify emotions from each modality. By integrating these predictions, we aim to provide a holistic understanding of a user's emotional state and assess potential risks. The system further analyzes the collected emotion data to identify trends, patterns, and indicators of emotional well-being and suicide risk. Visualizations such as time-series plots, distribution charts, and stacked area charts are employed to represent the emotional dynamics over time.Based on the analysis; the system generates personalized recommendations and alerts for users exhibiting signs of distress or emotional instability, including potential suicide risk factors. The goal is to promote proactive well-being management and early intervention. It is crucial to emphasize that this system is a tool for early detection and should be used in conjunction with professional mental health care.

Ah-Aloud Method to Comprehend Time-Series Emotion Observation During Gameplay: An Initial Investigation with Japanese Speakers

Emotional dimensions significantly impact user experience (UX) and are essential in assessing interactions in entertainment systems. However, relying solely on subjective post-event indicators fails to capture dynamic emotional changes during the system experience. Kawashima and Watanabe introduced the "ah-aloud" method, enabling real-time emotion observation by having participants vocalize "ah" during system experiences. However, this method remains conceptual and preliminary, lacking in specific experimental procedures, analysis methods, and utilization guidelines. Our study delineates the requirements for practical "ah-aloud" application and validates its potential for game experience assessment. Consequently, the "ah-aloud" method demonstrates the potential for observing time-series emotional changes during gameplay. Based on these findings, we deliberated its utility for system evaluation and reliability as an evaluation experiment.

أنظمة التفاعل التكيفي بين الإنسان والحاسوب مع التعرف على عواطف المستخدم والتنبؤ بالاستجابة المستندة إلى GRU

Human-Computer Interaction (HCI) has evolved to incorporate sophisticated systems capable of recognizing and responding to user emotions, enhancing the user experience by making interactions more intuitive and engaging. This paper explores the development of adaptive HCI systems utilizing Gated Recurrent Unit (GRU) neural networks for emotion recognition and response prediction. The core objective is to create interfaces that dynamically adjust based on the user's emotional state, thereby improving usability and satisfaction across various applications, including healthcare, education, and customer service. GRU-based models are particularly effective for this task due to their ability to handle sequential data and capture temporal dependencies inherent in emotional expressions. By processing multimodal inputs such as facial expressions, voice intonations, text, and physiological signals, these systems can accurately detect and predict emotions in real-time. The DEAP dataset, which includes EEG and other physiological recordings along with self-assessed emotional ratings, serves as a foundational resource for training and validating these models. The proposed approach involves pre-processing the physiological data using techniques like min-max normalization to ensure consistency and stability during model training. The GRU models then learn to map these inputs to corresponding emotional states, leveraging their memory capabilities to retain and interpret temporal patterns. The adaptive nature of these systems enables them to provide personalized responses, such as adjusting the interface layout or offering support when negative emotions are detected. The implementation of GRU-based emotion recognition and response prediction in HCI systems holds significant potential for enhancing user interactions by making them more responsive and emotionally intelligent. This paper demonstrates the effectiveness of GRU models in real-time emotion monitoring and highlights their applications in creating more adaptive and empathetic technology interfaces. The proposed model is implemented in Python and has an accuracy of about 99.12% which is higher than other existing methods. Keywords: Human-Computer Interaction (HCI), Gated Recurrent Unit (GRU), Neural Networks, Emotional Detection.

Towards Emotionally Intelligent Virtual Environments: Classifying Emotions through a Biosignal-Based Approach

This paper introduces a novel method for emotion classification within virtual reality (VR) environments, which integrates biosignal processing with advanced machine learning techniques. It focuses on the processing and analysis of electrocardiography (ECG) and galvanic skin response (GSR) signals, which are established indicators of emotional states. To develop a predictive model for emotion classification, we extracted key features, i.e., heart rate variability (HRV), morphological characteristics, and Hjorth parameters. We refined the dataset using a feature selection process based on statistical techniques to optimize it for machine learning applications. The model achieved an accuracy of 97.78% in classifying emotional states, demonstrating that by accurately identifying and responding to user emotions in real time, VR systems can become more immersive, personalized, and emotionally resonant. Ultimately, the potential applications of this method are extensive, spanning various fields. Emotion recognition in education would allow further implementation of adapted learning environments through responding to the current emotional states of students, thereby fostering improved engagement and learning outcomes. The capability for emotion recognition could be used by virtual systems in psychotherapy to provide more personalized and effective therapy through dynamic adjustments of the therapeutic content. Similarly, in the entertainment domain, this approach could be extended to provide the user with a choice regarding emotional preferences for experiences. These applications highlight the revolutionary potential of emotion recognition technology in improving the human-centric nature of digital experiences.

Optimizing Interactive Online Education Through Advanced Multimodal Emotion Recognition Techniques

Driven by the global pandemic, the demand for online education has significantly increased, making it crucial to enhance the interactive experience of online teaching to improve student learning outcomes and engagement. In this context, we have designed a model based on Particle Swarm Optimization (PSO) that combines the strengths of LSTNet in handling long-term dependencies and the capabilities of the Prophet model in trend and seasonality modeling. Our PSO-optimized LSTNet-Prophet model outperformed baseline models by 10% in accuracy and 12% in F-1 score, as shown by experiments conducted on the CMU-MOSEI dataset. Additionally, we have implemented a real-time emotion monitoring system capable of analyzing students' emotional states in real-time and providing feedback to teachers, aiding them in promptly adjusting teaching strategies, thereby improving teaching quality and interaction effectiveness. Our method achieved an emotion recognition accuracy of 82.42% and an F-1 score of 82.31%, demonstrating its effectiveness and robustness.

Kaleidoscope: Detecting the Tone of the Room

We aimed to develop a software tool that can determine the “tone of a room” in virtual meetings. Our prototype, Kaleidoscope, uses preexisting machine learning and deep learning libraries to analyze the expressed emotions from individuals in an online meeting, and turn that data into an overall analysis of group tone through novel analytic tools. Results from human participants, asked to identify emotions observed from online meeting videos, are used to validate and refine the prototype. The Kaleidoscope prototype currently achieves an overall accuracy greater than 70%, more than double that of human-to-human agreement; and demonstrates the unique capability of identifying the group tone of a meeting. Future extensions could provide facilitators real-time emotion tracking and group tone assessment, paving the way for technology-assisted interventions to combat group polarization and other problematic group dynamics.

Developing A Mobile Application to Determine the Psychological Wellness of University Students

of university students. Material and Methods: Using an innovative and interdisciplinary approach, this study follows a rigorous methodology from September 2022 to June 2023. The development of the mobile application followed a systematic and structured process to ensure functionality, usability, and reliability. Throughout all stages of development, professional support was sought from information technology to ensure the technical robustness, reliability, and effectiveness of the application. The study embraces four developmental stages including needs identification, technical development, intervention design, and promotion. These stages ensure a student-centered approach, while the application itself offers insights into depression, anxiety, stress, relationship violence attitudes, addiction, Internet addiction, sleep quality, and eating disorders. Results: A unique and all-inclusive mobile application was created to assess and enhance the psychological well-being of university students. With the real-time emotion monitoring feature of the application, students may keep an eye on their present emotional states and develop self-awareness. If long-term negative emotions are detected, the early warning system is activated and implemented. It provides uninterrupted referrals to qualified professionals for immediate response and support in emergency situations and an enhanced feedback mechanism for user complaints and suggestions. It also provides sensitive evaluation and triage processes by creating an anonymous system record to provide instant support, when necessary. The application also includes stress management guidance for students. Conclusion: This application provides real-time data that offers mental health professionals a comprehensive overview of students' psychological states.

Emotion Detection from EEG Signals Using Machine Deep Learning Models.

Detecting emotions is a growing field aiming to comprehend and interpret human emotions from various data sources, including text, voice, and physiological signals. Electroencephalogram (EEG) is a unique and promising approach among these sources. EEG is a non-invasive monitoring technique that records the brain's electrical activity through electrodes placed on the scalp's surface. It is used in clinical and research contexts to explore how the human brain responds to emotions and cognitive stimuli. Recently, its use has gained interest in real-time emotion detection, offering a direct approach independent of facial expressions or voice. This is particularly useful in resource-limited scenarios, such as brain-computer interfaces supporting mental health. The objective of this work is to evaluate the classification of emotions (positive, negative, and neutral) in EEG signals using machine learning and deep learning, focusing on Graph Convolutional Neural Networks (GCNN), based on the analysis of critical attributes of the EEG signal (Differential Entropy (DE), Power Spectral Density (PSD), Differential Asymmetry (DASM), Rational Asymmetry (RASM), Asymmetry (ASM), Differential Causality (DCAU)). The electroencephalography dataset used in the research was the public SEED dataset (SJTU Emotion EEG Dataset), obtained through auditory and visual stimuli in segments from Chinese emotional movies. The experiment employed to evaluate the model results was "subject-dependent". In this method, the Deep Neural Network (DNN) achieved an accuracy of 86.08%, surpassing SVM, albeit with significant processing time due to the optimization characteristics inherent to the algorithm. The GCNN algorithm achieved an average accuracy of 89.97% in the subject-dependent experiment. This work contributes to emotion detection in EEG, emphasizing the effectiveness of different models and underscoring the importance of selecting appropriate features and the ethical use of these technologies in practical applications. The GCNN emerges as the most promising methodology for future research.