Real-time Emotion Recognition Research Articles

Enhancing Real-Time Emotion Recognition in Classroom Environments Using Convolutional Neural Networks: A Step Towards Optical Neural Networks for Advanced Data Processing

In contemporary academic settings, end-of-semester student feedback on a lecturer’s teaching abilities often fails to provide a comprehensive, real-time evaluation of their proficiency, and becomes less relevant with each new cohort of students. To address these limitations, an innovative feedback method has been proposed, utilizing image processing algorithms to dynamically assess the emotional states of students during lectures by analyzing their facial expressions. This real-time approach enables lecturers to promptly adapt and enhance their teaching techniques. Recognizing and engaging with emotionally positive students has been shown to foster better learning outcomes, as their enthusiasm actively stimulates cognitive engagement and information analysis. The purpose of this work is to identify emotions based on facial expressions using a deep learning model based on a convolutional neural network (CNN), where facial recognition is performed using the Viola–Jones algorithm on a group of students in a learning environment. The algorithm encompasses four key steps: image acquisition, preprocessing, emotion detection, and emotion recognition. The technological advancement of this research lies in the proposal to implement photonic hardware and create an optical neural network which offers unparalleled speed and efficiency in data processing. This approach demonstrates significant advancements over traditional electronic systems in handling computational tasks. An experimental validation was conducted in a classroom with 45 students, demonstrating that the level of understanding in the class as predicted was 43–62.94%, and the proposed CNN algorithm (facial expressions detection) achieved an impressive 83% accuracy in understanding students’ emotional states. The correlation between the CNN deep learning model and the students’ feedback was 91.7%. This novel approach opens avenues for the real-time assessment of students’ engagement levels and the effectiveness of the learning environment, providing valuable insights for ongoing improvements in teaching practices.

AutYOLO-ATT: an attention-based YOLOv8 algorithm for early autism diagnosis through facial expression recognition

Autism Spectrum Disorder (ASD) is a developmental condition resulting from abnormalities in brain structure and function, which can manifest as communication and social interaction difficulties. Conventional methods for diagnosing ASD may not be effective in the early stages of the disorder. Hence, early diagnosis is crucial to improving the patient's overall health and well-being. One alternative and effective method for early autism diagnosis is facial expression recognition since autistic children typically exhibit distinct facial expressions that can aid in distinguishing them from other children. This paper provides a deep convolutional neural network (DCNN)-based real-time emotion recognition system for autistic kids. The proposed system is designed to identify six facial emotions, including surprise, delight, sadness, fear, joy, and natural, and to assist medical professionals and families in recognizing facial expressions in autistic children for early diagnosis and intervention. In this study, an attention-based YOLOv8 (AutYOLO-ATT) algorithm for facial expression recognition is proposed, which enhances the YOLOv8 model's performance by integrating an attention mechanism. The proposed method (AutYOLO-ATT) outperforms all other classifiers in all metrics, achieving a precision of 93.97%, recall of 97.5%, F1-score of 92.99%, and accuracy of 97.2%. These results highlight the potential of the proposed method for real-world applications, particularly in fields where high accuracy is essential.

Mist–fog-assisted real-time emotion recognition using deep transfer learning framework for smart city 4.0

Face emotion recognition (FER) has found several applications across numerous industries. One such application area lies in the field of human–computer interaction, where it can improve user experiences by allowing systems to adjust based on users’ emotional states by leveraging the capabilities of the Internet of Things (IoT) to facilitate more responsive interactions. The increasing demand for precise and immediate emotion analysis across various applications drives the need for FER utilizing deep transfer learning (DTL). In this study, a novel Mist-Fog framework for real-time FER is introduced, which employs a DTL algorithm called Mobile-Net for quick and easy image classification on mobile and embedded devices. It addresses the challenges of operating deep neural networks (DNN) under limits on power and processing resources. By utilizing depth-wise separable convolutions, the MobileNet algorithm reduces computational complexity while maintaining reasonable accuracy on the FER-2013 dataset. A Fog-based architecture, in which a mist layer comprising edge devices like Raspberry-Pi (RPi), an ARM-based single-board computer, was considered for a smart city scenario. Further, the M/M/1/K queuing system was considered, which follows the First-Come-First-Serve (FCFS) scheduling criteria and a Markov Modulated Poisson Process (MMPP) arrival rate to evaluate the system’s performance. The effectiveness of this novel approach towards industrial computer vision paradigms is assessed pertaining to different performance criteria.

EMOJIFY

Emoticons have become a ubiquitous part of online communication, but their limitations hinder the expression of nuanced emotions. This project proposes "Emojify," a system that leverages deep learning for real-time emotion recognition through video input. By employing techniques like pre-processing, face detection, and Convolutional Neural Networks (CNNs), Emojify analyzes facial expressions and generates corresponding emojis. This approach goes beyond traditional emoticons, dynamically reflecting emotional intent and enriching virtual communication. The project explores the potential for entertainment applications and user-friendly emoji creation tools with encryption and decryption methods. Emojify holds promise for bridging the emotional gap in virtual interactions, fostering a more expressive and engaging online experience.

Facial Emotion Detection: Applications, Advancements, and Implications in Human-Computer Interaction

Abstract: The YOLO Version 8 algorithm, considered for its real-time object detection abilities, is implemented to demonstrate the face emotion detection. We integrate deep neural network analysis, accurate landmark detection, and face image preprocessing to achieve enhanced effectiveness and precision in real-time emotion recognition. This is made possible by YOLO V8's sophisticated architecture, which effectively processes visual input. Robust performance in various demographic and cultural contexts is guaranteed by extensive training on a variety of datasets. Comprehensive evaluations indicate that the system surpasses existing methodologies in terms of precision, efficiency, and adaptability. Anticipated advancements in emotion-aware technologies promise far-reaching implications, with potential applications spanning sentiment analysis, interactive computing, and mental well-being monitoring.

Research on Speech Emotion Recognition System Using Machine Learning

Speech Emotion Recognition (SER) remains a hot topic in the domain of affective computing, drawing considerable research interest. Its growing potential, advancements in algorithms, and real-world applications contribute to this interest. Human speech carries paralinguistic cues that can be quantitatively represented through features like pitch, intensity, and Mel-Frequency Cepstral Coefficients (MFCC). SER typically involves three main stages: data processing, feature extraction/selection, and classification based on emotional features present. These steps, tailored to the unique attributes of human speech, make machine learning methods a fitting choice for SER implementation. While recent studies in affective computing leverage various ML techniques for SER tasks, few delve into the techniques aiding these core steps. Moreover, the challenges within these stages and cutting-edge approaches addressing them often receive limited discussion or are overlooked in these works.This project introduces a pioneering Speech Emotion Recognition System leveraging BiLSTM Algorithm and Image Processing techniques, developed for implementation on the Raspberry Pi platform using Python. The system analyses recorded audio input, employing three LEDs for mood detection and interfacing with Raspberry Pi programming facilitated by an SD card. This abstract outlines an all-encompassing approach for real-time emotion recognition via audio analysis, catering to varied applications in emotional AI and human-computer interaction. KEYWORDS: Speech Emotion Recognition (SER), Machine Learning, Image Processing, Raspberry Pi.

Real-Time Emotion Recognition and its Effects in a Learning Environment.

The purpose of the article is to understand the current state of both the technology and the implementation of emotion recognition in the educational environment. The goal is to obtain detailed information about the current state of emotion recognition technology and how its practical use is being carried out in educational settings. In this line, it examines the proposals from publications over the last 10 years on the advancement of technology for emotion recognition in education. A total of 1,347 studies were obtained and 43 were included in the review for analysis and discussion. The article demonstrates how the number of studies has increased in recent years, with a higher frequency in online learning. Furthermore, according to the Technological Readiness Level, despite the growing interest in emotion recognition in the educational environment, its implementation is still far from becoming a reality. Most of the research has been conducted from a theoretical perspective and none of them has been fully developed and implemented in the classroom. In addition, many of the studies analysed have not tested the validity of their findings.

A Comprehensive Evaluation of Features and Simple Machine Learning Algorithms for Electroencephalographic-Based Emotion Recognition

The study of electroencephalographic (EEG) signals has gained popularity in recent years because they are unlikely to intentionally fake brain activity. However, the reliability of the results is still subject to various noise sources and potential inaccuracies inherent to the acquisition process. Analyzing these signals involves three main processes: feature extraction, feature selection, and classification. The present study extensively evaluates feature sets across domains and their impact on emotion recognition. Feature selection improves results across the different domains. Additionally, hybrid models combining features from various domains offer a superior performance when applying the public DEAP dataset for emotion classification using EEG signals. Time, frequency, time–frequency, and spatial domain attributes and their combinations were analyzed. The effectiveness of the input vectors for the classifiers was validated using SVM, KNN, and ANN, which are simple classification algorithms selected for their widespread use and better performance in the state of the art. The use of simple machine learning algorithms makes the findings particularly valuable for real-time emotion recognition applications where the computational resources and processing time are often limited. After the analysis stage, feature vector combinations were proposed to identify emotions in four quadrants of the valence–arousal representation space using the DEAP dataset. This research achieved a classification accuracy of 96% using hybrid features in the four domains and the ANN classifier. A lower computational cost was obtained in the frequency domain.

Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface

Human affects such as emotions, moods, feelings are increasingly being considered as key parameter to enhance the interaction of human with diverse machines and systems. However, their intrinsically abstract and ambiguous nature make it challenging to accurately extract and exploit the emotional information. Here, we develop a multi-modal human emotion recognition system which can efficiently utilize comprehensive emotional information by combining verbal and non-verbal expression data. This system is composed of personalized skin-integrated facial interface (PSiFI) system that is self-powered, facile, stretchable, transparent, featuring a first bidirectional triboelectric strain and vibration sensor enabling us to sense and combine the verbal and non-verbal expression data for the first time. It is fully integrated with a data processing circuit for wireless data transfer allowing real-time emotion recognition to be performed. With the help of machine learning, various human emotion recognition tasks are done accurately in real time even while wearing mask and demonstrated digital concierge application in VR environment.

Multimodal Real-Time patient emotion recognition system using facial expressions and brain EEG signals based on Machine learning and Log-Sync methods

Human Machine Interface (HMI) depends on emotion detection, especially for hospitalized patients. The emergence of the fourth industrial revolution (4IR) has heightened the interest in emotional intelligence in human–computer interaction (HCI). This work employs electroencephalography (EEG), an optical flow algorithm, and machine learning to create a multimodal intelligent real-time emotion recognition system. The objective is to assist hospitalized patients, disabled (deaf, mute, and bedridden) individuals, and autistic youngsters in expressing their authentic feelings. We fed our multimodal feature fusion vector to a classifier with long short-term memory (LSTM). We distinguished six fundamental emotions: anger, disgust, fear, sadness, joy, and surprise. The fusion feature vector was created utilizing the patient's geometric facial characteristics and EEG inputs. Utilizing 14 EEG inputs, we used four-band relative power channels, namely alpha (8–13 Hz), beta (13–30 Hz), gamma (30–49 Hz), and theta (4–8 Hz). We achieved a maximum recognition rate of 90.25 percent using just facial landmarks and 87.25 percent using only EEG data. When both facial and EEG streams were examined, we achieved 99.3 percent accuracy in a multimodal method.

Real-time facial emotion recognition model based on kernel autoencoder and convolutional neural network for autism children

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that is characterized by abnormalities in the brain, leading to difficulties in social interaction and communication, as well as learning and attention. Early diagnosis of ASD is challenging as it mainly relies on detecting abnormalities in brain function, which may not be evident in the early stages of the disorder. Facial expression analysis has shown promise as an alternative and efficient solution for early diagnosis of ASD, as children with ASD often exhibit distinctive patterns that differentiate them from typically developing children. Assistive technology has emerged as a crucial tool in improving the quality of life for individuals with ASD. In this study, we developed a real-time emotion identification system to detect the emotions of autistic children in case of pain or anger. The emotion recognition system consists of three stages: face identification, facial feature extraction, and feature categorization. The proposed system can detect six facial emotions: anger, fear, joy, natural, sadness, and surprise. To achieve high-performance accuracy in classifying the input image efficiently, we proposed a deep convolutional neural network (DCNN) architecture for facial expression recognition. An autoencoder was used for feature extraction and feature selection, and a pre-trained model (ResNet, MobileNet, and Xception) was applied due to the size of the dataset. The Xception model achieved the highest performance, with an accuracy of 0.9523%, sensitivity of 0.932, specificity of 0.9421, and AUC of 0.9134%. The proposed emotion detection framework leverages fog and IoT technologies to reduce latency for real-time detection with fast response and location awareness. Using fog computing is particularly useful when dealing with big data. Our study demonstrates the potential of using facial expression analysis and deep learning algorithms for real-time emotion recognition in autistic children, providing medical experts and families with a valuable tool for improving the quality of life for individuals with ASD.

Real-time EEG-based emotion recognition for neurohumanities: perspectives from principal component analysis and tree-based algorithms.

Within the field of Humanities, there is a recognized need for educational innovation, as there are currently no reported tools available that enable individuals to interact with their environment to create an enhanced learning experience in the humanities (e.g., immersive spaces). This project proposes a solution to address this gap by integrating technology and promoting the development of teaching methodologies in the humanities, specifically by incorporating emotional monitoring during the learning process of humanistic context inside an immersive space. In order to achieve this goal, a real-time emotion recognition EEG-based system was developed to interpret and classify specific emotions. These emotions aligned with the early proposal by Descartes (Passions), including admiration, love, hate, desire, joy, and sadness. This system aims to integrate emotional data into the Neurohumanities Lab interactive platform, creating a comprehensive and immersive learning environment. This work developed a ML, real-time emotion recognition model that provided Valence, Arousal, and Dominance (VAD) estimations every 5 seconds. Using PCA, PSD, RF, and Extra-Trees, the best 8 channels and their respective best band powers were extracted; furthermore, multiple models were evaluated using shift-based data division and cross-validations. After assessing their performance, Extra-Trees achieved a general accuracy of 94%, higher than the reported in the literature (88% accuracy). The proposed model provided real-time predictions of VAD variables and was adapted to classify Descartes' six main passions. However, with the VAD values obtained, more than 15 emotions can be classified (reported in the VAD emotion mapping) and extend the range of this application.

Real-Time Emotion Recognition using Deep Facial Expression Analysis on Mobile Devices

Numerous applications in human-computer interaction, healthcare, and other fields have been made possible by the growth of mobile devices, which has created new opportunities for real-time emotion recognition. This research introduces a novel method for mobile device-based deep facial expression analysis for real-time emotion recognition. In order to execute accurate and efficient emotion recognition directly on the device, our technology makes use of the computational capacity of contemporary smartphones. This eliminates the need for cloud-based processing and ensures user privacy. For mobile platforms, we use a deep learning architecture that is optimised for both speed and accuracy. A lightweight convolutional neural network (CNN) for facial feature extraction and a recurrent neural network (RNN) for temporal emotion modelling are the main elements of our system. Our system accurately detects and categorises a variety of emotions, including joy, sadness, rage, surprise, and more by processing video frames from the device's camera feed in real-time. We carried out extensive trials on a variety of datasets to assess our methodology, attaining state-of-the-art accuracy with minimal processing cost. Through a variety of applications, including emotion-aware virtual assistants, mental health tracking tools, and immersive gaming experiences, we show how useful our technology is. This paper makes a contribution to the burgeoning field of mobile-based emotion recognition by providing a strong and effective solution that enables researchers and developers to produce ground-breaking software that can better comprehend and react to human emotions while protecting data privacy and guaranteeing real-time performance on mobile devices.

Leveraging a Hybrid Deep Learning Architecture for Efficient Emotion Recognition in Audio Processing

This paper presents a novel hybrid deep learning architecture for emotion recognition from speech signals, which has garnered significant interest in recent years due to its potential applications in various fields such as healthcare, psychology, and entertainment. The proposed architecture combines modified ResNet-34 and RoBERTa models to extract meaningful features from speech signals and classify them into different emotion categories. The model is evaluated on five standard emotion recognition datasets, including RAVDESS, EmoDB, SAVEE, CREMA-D, and TESS, and achieves state-of-the-art performance on all datasets. The experimental results show that the proposed hybrid architecture outperforms existing emotion recognition models, achieving high accuracy and F1 scores for emotion classification. The proposed architecture is promising for real-time emotion recognition applications and can be applied in various domains such as speech-based emotion recognition systems, human-computer interaction, and virtual assistants.

Emotion recognition by skeleton-based spatial and temporal analysis

This study introduces an automatic emotion recognition system (AER) focusing on skeletal-based kinematic datasets for enhanced human–computer interaction. Departing from conventional approaches, it achieves real-time emotion recognition in real-life situations. The dataset covers seven emotions and undergoes assessment by eight diverse machine and deep learning algorithms. A thorough investigation is undertaken by varying window sizes and data states, including raw positions and feature-extracted data. The findings imply that incorporating advanced techniques like joint-related feature extraction and robust classifier models yields promising outcomes. Dataset augmentation via varying window sizes enriches insights into real-world scenarios. Evaluations exhibit classification accuracy surpassing 99% for small windows, 94% for medium, and exceeding 88% for larger windows, thereby confirming the robust nature of the approach. Furthermore, we highlight window size's impact on emotion detection and the benefits of combining coordinate axes for efficiency and accuracy. The analysis intricately examines the contributions of features at both the joint and axis levels, assisting in making well-informed selections. The study's contributions include carefully curated datasets, transparent code, and models, all of which ensure the possibility of replication. The paper establishes a benchmark that bridges theory and practicality, solidifying the proposed approach's effectiveness in balancing accuracy and efficiency. By pioneering advanced AER through kinematic data, it sets a new standard for efficacy while driving seamless human–computer interaction through rigorous analysis and strategic design.

SMFNM: Semi-supervised multimodal fusion network with main-modal for real-time emotion recognition in conversations

Real-time emotion recognition in conversations (ERC), which relies on only the historical utterances to achieve ERC, has recently gained increasing attention due to its significance in providing real-time empathetic services. Although utilizing multimodal information can mitigate the issues of unimodal approaches, few real-time ERC studies consider the differences in representation ability of different modalities and explore comprehensive conversational context from different perspectives based on different structures. Furthermore, the heavy annotation cost makes it difficult to collect sufficient labeled data, which also limits the performance of current supervised ERC approaches. To address these issues, we propose a novel framework SMFNM for real-time ERC, which integrates semi-supervised learning with multimodal fusion under the guidance of main-modal. Specifically, SMFNM utilizes additional unlabeled data to extract high-quality intra-modal representations, and implements cross-modal interaction to capture complementary information to enhance the audio representations. Then SMFNM employs the directed acyclic graph and the Gated Recurrent Units for exploring more accurate conversational context from both the multimodal and main-modal perspectives, respectively. Finally, these two types of contextual features are fused for emotion identification. Extensive experiments on benchmark datasets (i.e., IEMOCAP (4-way), IEMOCAP (6-way) and MELD) demonstrate the effectiveness, superiority and rationality of our SMFNM.

Development of Real-Time Emotion Recognition System Using Facial Expressions

This research presents a real-time emotion recognition system that combines human-friendly machine interaction with picture processing. For many years, facial detection has been available. Moving further, it is possible to simulate the emotions that people express on their faces and experience in their brains through the use of video, electric signals, or image forms. Since it is hard for computers to detect emotions from images or videos and a difficult task for the human eye, machine emotion detection requires a variety of image processing approaches for feature extraction. The approach proposed in this paper consists of two primary processes: facial expression recognition (FER) and face detection. The experimental investigation of facial emotion recognition is the main topic of this study. An emotion detection system's workflow consists of face detection, feature extraction, pre-processing, classification, and image acquisition. The emotion identification system uses the Haar cascade algorithm, an object detection algorithm, to recognize faces in an image or a real-time video, and the KNN Classifier for image classification in order to identify such emotions. Using the webcam to capture real-time photos, this system operates. The goal of this research is to develop an automatic facial expression recognition system that can recognize various emotions. Based on these studies, the system may be able to distinguish between a number of people who are fearful, furious, shocked, sad, or pleased, among other emotions.

SGCRNN: A ChebNet-GRU fusion model for eeg emotion recognition

The paper proposes a deep learning model based on Chebyshev Network Gated Recurrent Units, which is called Spectral Graph Convolution Recurrent Neural Network, for multichannel electroencephalogram emotion recognition. First, in this paper, an adjacency matrix capturing the local relationships among electroencephalogram channels is established based on the cosine similarity of the spatial locations of electroencephalogram electrodes. The training efficiency is improved by utilizing the computational speed of the cosine distance. This advantage enables our method to have the potential for real-time emotion recognition, allowing for fast and accurate emotion classification in real-time application scenarios. Secondly, the spatial and temporal dependence of the Spectral Graph Convolution Recurrent Neural Network for capturing electroencephalogram sequences is established based on the characteristics of the Chebyshev network and Gated Recurrent Units to extract the spatial and temporal features of electroencephalogram sequences. The proposed model was tested on the publicly accessible dataset DEAP. Its average recognition accuracy is 88%, 89.5%, and 89.7% for valence, arousal, and dominance, respectively. The experiment results demonstrated that the Spectral Graph Convolution Recurrent Neural Network method performed better than current models for electroencephalogram emotion identification. This model has broad applicability and holds potential for use in real-time emotion recognition scenarios.

Real-Time EEG-Based Emotion Recognition.

Most studies have demonstrated that EEG can be applied to emotion recognition. In the process of EEG-based emotion recognition, real-time is an important feature. In this paper, the real-time problem of emotion recognition based on EEG is explained and analyzed. Secondly, the short time window length and attention mechanisms are designed on EEG signals to follow emotion change over time. Then, long short-term memory with the additive attention mechanism is used for emotion recognition, due to timely emotion updates, and the model is applied to the SEED and SEED-IV datasets to verify the feasibility of real-time emotion recognition. The results show that the model performs relatively well in terms of real-time performance, with accuracy rates of 85.40% and 74.26% on SEED and SEED-IV, but the accuracy rate has not reached the ideal state due to data labeling and other losses in the pursuit of real-time performance.

Physiological Signal-Based Real-Time Emotion Recognition Based on Exploiting Mutual Information with Physiologically Common Features

This paper proposes a real-time emotion recognition system that utilizes photoplethysmography (PPG) and electromyography (EMG) physiological signals. The proposed approach employs a complex-valued neural network to extract common features from the physiological signals, enabling successful emotion recognition without interference. The system comprises three stages: single-pulse extraction, a physiological coherence feature module, and a physiological common feature module. The experimental results demonstrate that the proposed method surpasses alternative approaches in terms of accuracy and the recognition interval. By extracting common features of the PPG and EMG signals, this approach achieves effective emotion recognition without mutual interference. The findings provide a significant advancement in real-time emotion analysis and offer a clear and concise framework for understanding individuals’ emotional states using physiological signals.