Emotion Recognition System Research Articles

Marathi Speech Emotion recognition using Deep Learning techniques.

Marathi Speech Emotion recognition using Deep Learning techniques.

Face and Speech Emotion Recognition System

Abstract: Emotions serve as the cornerstone of human communication, facilitating the expression of one's inner thoughts and feelings to others. Speech Emotion Recognition (SER) represents a pivotal endeavour aimed at deciphering the emotional nuances embedded within a speaker's voice signal. Universal emotions such as neutrality, anger, happiness, and sadness form the basis of this recognition process, allowing for the identification of fundamental emotional states. To achieve this, spectral and prosodic features are leveraged, each offering unique insights into the emotional content of speech. Spectral features, exemplified by the Mel Frequency Cepstral Coefficient (MFCC), provide a detailed analysis of the frequency distribution within speech signals, while prosodic features encompass elements like fundamental frequency, volume, pitch, speech intensity, and glottal parameters, capturing the rhythmic and tonal variations indicative of different emotional states. Through the integration of these features, SER systems can effectively simulate and classify a diverse range of emotional expressions, paving the way for enhanced human-computer interaction and communication technologies

A Systematic Review of Electroencephalography-Based Emotion Recognition of Confusion Using Artificial Intelligence

Confusion emotion in a learning environment can motivate the learner, but prolonged confusion hinders the learning process. Recognizing confused learners is possible; nevertheless, finding them requires a lot of time and effort. Due to certain restrictions imposed by the settings of an online learning environment, the recognition of confused students is a big challenge for educators. Therefore, novel technologies are necessary to handle such crucial difficulties. Lately, Electroencephalography (EEG)-based emotion recognition systems have been rising in popularity in the domain of Education Technology. Such systems have been utilized to recognize the confusion emotion of learners. Numerous studies have been conducted to recognize confusion emotion through this system since 2013, and because of this, a systematic review of the methodologies, feature sets, and utilized classifiers is a timely necessity. This article presents the findings of the review conducted to achieve this requirement. We summarized the published literature in terms of the utilized datasets, feature preprocessing, feature types for model training, and deployed classifiers in terms of shallow machine learning and deep learning-based algorithms. Moreover, the article presents a comparison of the prediction accuracies of the classifiers and illustrates the existing research gaps in confusion emotion recognition systems. Future study directions for potential research are also suggested to overcome existing gaps.

Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks

This study describes an AI model by leveraging advanced Convolutional Neural Networks (CNNs) to recognize affective states in real-world sports settings, particularly tennis matches. In contrast to prior studies that primarily utilized data acquired from actors and rudimentary statistical methods, the present research emphasizes the analysis of bodily expressions in real-life contexts, aiming for a more naturalistic representation of human emotions. Our CNN-based models demonstrate an accuracy rate of up to 68.9 %, outperforming or matching human observers in many instances. Intriguingly, both the machine learning models and human observers exhibited a shared propensity to more effectively identify negative affective states, which may be attributed to the more intense and straightforward expression of these states. These results not only advance the state of the art in affective state recognition but also pave the way for broader applications, including in healthcare and automotive safety sectors, thereby constituting a significant advancement in the development of sophisticated and universally applicable emotional recognition systems.

A Modified MFCC-Based Deep Learning Method for Emotion Classification from Speech

Speech, which is one of the most effective methods of communication, varies according to the emotions experienced by people and includes not only vocabulary but also information about emotions. With developing technologies, human-machine interaction is also improving. Emotional information to be extracted from voice signals is valuable for this interaction. For these reasons, studies on emotion recognition systems are increasing. In this study, sentiment analysis is performed using the Toronto Emotional Speech Set (TESS) created by University of Toronto. The voice data in the dataset is first preprocessed and then a new CNN-based deep learning method on it is compared. The voice files in the TESS dataset have been first obtained feature maps using the MFCC method, and then classification has been performed with this method based on the proposed neural network model. Separate models have been created with CNN and LSTM models for the classification process. The experiments show that the MFCC-applied CNN model achieves a better result with an accuracy of 99.5% than the existing methods for the classification of voice signals. The accuracy value of the CNN model shows that the proposed CNN model can be used for emotion classification from human voice data.

Customer Feedback Analysis Using Facial Emotion Recognition

Our lives are being significantly impacted by the rapid development of wireless technology and mobile gadgets on this day. The digital economy demands that services be developed almost instantly while also paying close attention to client feedback. It becomes difficult to manage and analyze the information gathered about products from customers. Also, everyone is not intended to provide clear feedback whether the product was satisfactory or not. It is a very difficult and time- consuming task to analyze the data collected manually. To proceed with the problem and through much research we came across a solution, an emotion recognition system which can overcome this situation in real time. Keywords:CNN, Machine Learning, NaiveBayes.

Emotion Recognition for Improving Online Learning Environments: A Systematic Review of the Literature

The increased intervention of computer vision in enhancing learning environments has intrigued modern literature on educational research. Affective tutoring system (ATS), student emotion recognition system (SERS), sentiment analysis, and multi-agent system (MAS) have significantly enhanced the online learning environment. Without preceding technologies, online learning environment growth was found to be redundant due to a lack of a corrective mechanism in marginalized viewpoints of tutor in noting facts for their learning environment. As a result, computer vision was found to enhance the online learning environment by providing tutors with enhanced monitoring ability and assistive measures. The following study incorporated a systematic review of 33 papers, efficiently reflecting technology usage in education. The study examined methods for extracting and analyzing emotional data, machine learning algorithms, flexibility to individual learners, camera quality, and ethical issues. This study sought to illuminate these systems' pros and cons. This research shows that emotion recognition systems can improve online learning. These technologies quantify and analyze students' emotional responses, helping educators improve teaching techniques and material. With real-time input on students' emotions, teachers can adjust their methods to keep students engaged and increase academic performance.

Instructor emotion recognition system using manta ray foraging algorithm for improving the content delivery in video lecture

Instructor emotion recognition system using manta ray foraging algorithm for improving the content delivery in video lecture

Linguistic based emotion analysis using softmax over time attention mechanism.

Recognizing the real emotion of humans is considered the most essential task for any customer feedback or medical applications. There are many methods available to recognize the type of emotion from speech signal by extracting frequency, pitch, and other dominant features. These features are used to train various models to auto-detect various human emotions. We cannot completely rely on the features of speech signals to detect the emotion, for instance, a customer is angry but still, he is speaking at a low voice (frequency components) which will eventually lead to wrong predictions. Even a video-based emotion detection system can be fooled by false facial expressions for various emotions. To rectify this issue, we need to make a parallel model that will train on textual data and make predictions based on the words present in the text. The model will then classify the type of emotions using more comprehensive information, thus making it a more robust model. To address this issue, we have tested four text-based classification models to classify the emotions of a customer. We examined the text-based models and compared their results which showed that the modified Encoder decoder model with attention mechanism trained on textual data achieved an accuracy of 93.5%. This research highlights the pressing need for more robust emotion recognition systems and underscores the potential of transfer models with attention mechanisms to significantly improve feedback management processes and the medical applications.

Synthesizing affective neurophysiological signals using generative models: A review paper

The integration of emotional intelligence in machines is an important step in advancing human–computer interaction. This demands the development of reliable end-to-end emotion recognition systems. However, the scarcity of public affective datasets presents a challenge. In this literature review, we emphasize the use of generative models to address this issue in neurophysiological signals, particularly Electroencephalogram (EEG) and Functional Near-Infrared Spectroscopy (fNIRS). We provide a comprehensive analysis of different generative models used in the field, examining their input formulation, deployment strategies, and methodologies for evaluating the quality of synthesized data. This review serves as a comprehensive overview, offering insights into the advantages, challenges, and promising future directions in the application of generative models in emotion recognition systems. Through this review, we aim to facilitate the progression of neurophysiological data augmentation, thereby supporting the development of more efficient and reliable emotion recognition systems.

Evaluation of Eye-Blinking Dynamics in Human Emotion Recognition Using Weighted Visibility Graph

Purpose: Designing an automated emotion recognition system using biosignals has become a hot and challenging issue in many fields, including human-computer interferences, robotics, and affective computing. Several algorithms have been proposed to characterize the internal and external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are habitually analyzed in a multi-modality system using classic statistical measures, and the evaluation of its dynamics has been neglected so far.
 Materials and Methods: This experiment intended to provide an innovative single-modality scheme for emotion classification using eye-blinking data. The dynamics of eye-blinking data have been characterized by weighted visibility graph-based indices. The extracted measures were then fed to the different classifiers, including support vector machine, decision tree, k-Nearest neighbor, Adaptive Boosting, and random subset to complete the process of classifying sad, happy, neutral, and fearful affective states. The scheme has been evaluated utilizing the available signals in the SEED-IV database.
 Results: The proposed framework provided significant performance in terms of recognition rates. The highest average recognition rates of > 90% were achieved using the decision tree.
 Conclusion: In brief, our results showed that eye-blinking data has the potential for emotion recognition. The present system can be extended for designing future affect recognition systems.

Convey: Developing a Facial Emotion Recognition System for Enhancing Interpersonal Interactions of Children with Autism Spectrum Disorders

Children with Autism Spectrum Disorder (ASD) oftentimes have difficulty in both interpersonal and intrapersonal interactions. Most children with ASD lack emotional intelligence and cannot properly comprehend social cues and responses of the people around them. These social impairments are causing them to have difficulty in maintaining relationships thus, to combat these issues, the researchers presented an application with Facial Emotion Recognition (FER) technology using the deep-learning approach namely Convolutional Neural Network (CNN). Using the FER allows the children to perceive the emotions of their interaction partner at face value. The model used in developing the application sits at an accuracy level of 98%. In addition to the FER technology, the research implements a kind of mood meter that enables the children to plot their mood and a dashboard that tracks the accumulated plots. The tracker lets the children convey their emotions as well as help their interaction partner to understand them better. Furthermore, the application includes a recommendation system that suggests various relaxation and meditation techniques that the child can use to manage outbursts or spontaneous emotional activity. The Convey mobile application aimed to empower children that has ASD and help them lead a normal life.

AFLEMP: Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data

Automated emotion recognition systems utilizing physiological signals are essential for affective computing and intelligent interaction. Combining the multiple physiological signals is more precise and effective in accurately assessing a person’s emotional state. These automated emotion recognition systems using conventional machine learning techniques require complete access to the physiological data for emotion state classification, compromising sensitive data privacy. Federated Learning (FL) resolves this issue by preserving the user’s privacy and sensitive physiological data while recognizing emotions. However, existing FL methods have limitations in handling data heterogeneity in the physiological data and do not measure communication efficiency and scalability. In response to these challenges, this paper proposes a unique novel framework called AFLEMP (Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data) integrating attention mechanism-based Transformer with an Artificial Neural Network (ANN) model. The framework reduces two types of data heterogeneity: (1) Variation Heterogeneity (VH) in multi-modal EEG, GSR, and ECG physiological signal data using attention mechanisms and (2) Imbalanced Data Heterogeneity (IDH) in the FL environment using scaled weighted federated averaging. This paper validates the proposed AFLEMP framework on two publicly available emotion datasets, AMIGOS and DREAMER, achieving an average accuracy of 88.30% and 84.10%, respectively. The proposed AFLEMP framework proves robust, scalable, and efficient in communication. AFLEMP is the first FL framework to propose for emotion recognition using multi-modal physiological signals while reducing data heterogeneity and outperforming existing FL methods.

Exploring contactless techniques in multimodal emotion recognition: insights into diverse applications, challenges, solutions, and prospects

In recent years, emotion recognition has received significant attention, presenting a plethora of opportunities for application in diverse fields such as human–computer interaction, psychology, and neuroscience, to name a few. Although unimodal emotion recognition methods offer certain benefits, they have limited ability to encompass the full spectrum of human emotional expression. In contrast, Multimodal Emotion Recognition (MER) delivers a more holistic and detailed insight into an individual's emotional state. However, existing multimodal data collection approaches utilizing contact-based devices hinder the effective deployment of this technology. We address this issue by examining the potential of contactless data collection techniques for MER. In our tertiary review study, we highlight the unaddressed gaps in the existing body of literature on MER. Through our rigorous analysis of MER studies, we identify the modalities, specific cues, open datasets with contactless cues, and unique modality combinations. This further leads us to the formulation of a comparative schema for mapping the MER requirements of a given scenario to a specific modality combination. Subsequently, we discuss the implementation of Contactless Multimodal Emotion Recognition (CMER) systems in diverse use cases with the help of the comparative schema which serves as an evaluation blueprint. Furthermore, this paper also explores ethical and privacy considerations concerning the employment of contactless MER and proposes the key principles for addressing ethical and privacy concerns. The paper further investigates the current challenges and future prospects in the field, offering recommendations for future research and development in CMER. Our study serves as a resource for researchers and practitioners in the field of emotion recognition, as well as those intrigued by the broader outcomes of this rapidly progressing technology.

Research on Speech Emotion Recognition System Using Machine Learning

Speech Emotion Recognition (SER) remains a hot topic in the domain of affective computing, drawing considerable research interest. Its growing potential, advancements in algorithms, and real-world applications contribute to this interest. Human speech carries paralinguistic cues that can be quantitatively represented through features like pitch, intensity, and Mel-Frequency Cepstral Coefficients (MFCC). SER typically involves three main stages: data processing, feature extraction/selection, and classification based on emotional features present. These steps, tailored to the unique attributes of human speech, make machine learning methods a fitting choice for SER implementation. While recent studies in affective computing leverage various ML techniques for SER tasks, few delve into the techniques aiding these core steps. Moreover, the challenges within these stages and cutting-edge approaches addressing them often receive limited discussion or are overlooked in these works.This project introduces a pioneering Speech Emotion Recognition System leveraging BiLSTM Algorithm and Image Processing techniques, developed for implementation on the Raspberry Pi platform using Python. The system analyses recorded audio input, employing three LEDs for mood detection and interfacing with Raspberry Pi programming facilitated by an SD card. This abstract outlines an all-encompassing approach for real-time emotion recognition via audio analysis, catering to varied applications in emotional AI and human-computer interaction. KEYWORDS: Speech Emotion Recognition (SER), Machine Learning, Image Processing, Raspberry Pi.

Spanish MEACorpus 2023: A multimodal speech–text corpus for emotion analysis in Spanish from natural environments

In human–computer interaction, emotion recognition provides a deeper understanding of the user’s emotions, enabling empathetic and effective responses based on the user’s emotional state. While deep learning models have improved emotion recognition solutions, it is still an active area of research. One important limitation is that most emotion recognition systems use only text as input, ignoring features such as voice intonation. Another limitation is the limited number of datasets available for multimodal emotion recognition. In addition, most published datasets contain emotions that are simulated by professionals and produce limited results in real-world scenarios. In other languages, such as Spanish, hardly any datasets are available. Therefore, our contributions to emotion recognition are as follows. First, we compile and annotate a new corpus for multimodal emotion recognition in Spanish (Spanish MEACorpus 2023), which contains 13.16 h of speech divided into 5129 segments labeled by considering Ekman’s six basic emotions. The dataset is extracted from YouTube videos in natural environments. Second, we explore several deep learning models for emotion recognition using text- and audio-based features. Third, we evaluate different multimodal techniques to build a multimodal recognition system that improves the results of unimodal models, achieving a Macro F1-score of 87.745%, using late fusion with concatenation strategy approach.

Decoding Human Emotions Through Multimodal Analysis Using Deep Learning

In the evolving landscape of emotion recognition, this study addresses the inherent shortcomings in existing technology. Recognizing the imperative role emotions play in human interaction, the need for a more nuanced and accurate multimodal emotion recognition system is evident. This research aims to refine multimodal emotion recognition by addressing existing challenges. The problem statement revolves around enhancing the accuracy and depth of emotion recognition through innovative methodologies. Leveraging Convolutional Neural Networks (CNN), Attention layers, and Bi-LSTM networks, the study utilizes a comprehensive approach. Data fusion involves combining audio extracted from video data, followed by fusion combinations of audio and text. The methodology incorporates SoftMax classifiers for feature recognition. The fusion of audio, text, and video data, coupled with the innovative use of CNN-LSTM and Attention layer networks, contributes to this success. The discussion interprets these results, highlighting the efficacy of the proposed approach in addressing the challenges of multimodal emotion recognition. The outcomes signify a substantial advancement in the field, providing a more comprehensive understanding of emotional expressions.

Emotion Modeling in Speech Signals: Discrete Wavelet Transform and Machine Learning Tools for Emotion Recognition System

Speech emotion recognition (SER) is a challenging task due to the complex and subtle nature of emotions. This study proposes a novel approach for emotion modeling using speech signals by combining discrete wavelet transform (DWT) with linear prediction coding (LPC). The performance of various classifiers, including support vector machine (SVM), K-Nearest Neighbors (KNN), Efficient Logistic Regression, Naive Bayes, Ensemble, and Neural Network, was evaluated for emotion classification using the EMO-DB dataset. Evaluation metrics such as area under the curve (AUC), average prediction accuracy, and cross-validation techniques were employed. The results indicate that KNN and SVM classifiers exhibited high accuracy in distinguishing sadness from other emotions. Ensemble methods and Neural Networks also demonstrated strong performance in sadness classification. While Efficient Logistic Regression and Naive Bayes classifiers showed competitive performance, they were slightly less accurate compared to other classifiers. Furthermore, the proposed feature extraction method yielded the highest average accuracy, and its combination with formants or wavelet entropy further improved classification accuracy. On the other hand, Efficient Logistic Regression exhibited the lowest accuracies among the classifiers. The uniqueness of this study was that it investigated a combined feature extraction method and integrated them to compare with various forms of combinations. However, the purposes of the investigation include improved performance of the classifiers, high effectiveness of the system, and the potential for emotion classification tasks. These findings can guide the selection of appropriate classifiers and feature extraction methods in future research and real-world applications. Further investigations can focus on refining classifiers and exploring additional feature extraction techniques to enhance emotion classification accuracy.

Student's Emotion Recognition using Multimodality and Deep Learning

The goal of emotion detection is to find and recognise emotions in text, speech, gestures, facial expressions, and more. This paper proposes an effective multimodal emotion recognition system based on facial expressions, sentence-level text, and voice. Using public datasets, we examine face expression image classification and feature extraction. The Tri-modal fusion is used to integrate the findings and to provide the final emotion. The proposed method has been verified in classroom students, and the feelings correlate with their performance. This method categorizes students' expressions into seven emotions: happy, surprise, sad, fear, disgust, anger, and contempt. Compared to the unimodal models, the suggested multimodal network design may reach up to 65% accuracy. The proposed method can detect negative feelings such as boredom or loss of interest in the learning environment.

Deep Learning-Based Wearable Ear-EEG Emotion Recognition System With Superlets-Based Signal-to-Image Conversion Framework

Deep Learning-Based Wearable Ear-EEG Emotion Recognition System With Superlets-Based Signal-to-Image Conversion Framework