Deep Learning Based Emotion Recognition Research Articles

Deep Learning-based Emotion Recognition Algorithms in Music Performance

In the realm of artificial intelligence and musicology, emotion recognition in music performance has emerged as a pivotal area of research. This paper introduces EmoTrackNet, an integrated deep learning framework that combines sparse attention networks, enhanced one-dimensional residual Convolutional Neural Networks (CNNs) with an improved Inception module, and Gate Recurrent Units (GRUs). The synergy of these technologies aims to decode complex emotional cues embedded in music. Our methodology revolves around leveraging the sparse attention network to efficiently process temporal sequences, thereby capturing the intricate dynamics of musical pieces. The incorporation of the 1D residual CNN with an upgraded Inception module facilitates the extraction of nuanced features from audio signals, encompassing a broad spectrum of musical tones and textures. The GRU component further refines the model’s capability to retain and process sequential information over longer timeframes, essential for understanding evolving emotional expressions in music. We evaluated EmoTrackNet on the Soundtrack dataset a comprehensive collection of music pieces annotated with emotional labels. The results demonstrated remarkable improvements in the accuracy of emotion recognition, outperforming existing models. This enhanced performance can be attributed to the integrated approach, which efficiently combines the strengths of each component, leading to a more robust and sensitive emotion detection system. EmoTrackNet’s novel architecture and promising results pave the way for new avenues in musicology, particularly in understanding and interpreting the emotional depth of musical performances. This framework not only contributes significantly to the field of music emotion recognition but also has potential applications in music therapy, entertainment, and interactive media where emotional engagement is key.

Explainable feature selection and deep learning based emotion recognition in virtual reality using eye tracker and physiological data.

Emotional recognition is a way of detecting, evaluating, interpreting, and responding to others' emotional states and feelings, which might range from delight to fear to disgrace. There is increasing interest in the domains of psychological computing and human-computer interface (HCI), especially Emotion Recognition (ER) in Virtual Reality (VR). Human emotions and mental states are effectively captured using Electroencephalography (EEG), and there has been a growing need for analysis in VR situations. In this study, we investigated emotion recognition in a VR environment using explainable machine learning and deep learning techniques. Specifically, we employed Support Vector Classifiers (SVC), K-Nearest Neighbors (KNN), Logistic Regression (LR), Deep Neural Networks (DNN), DNN with flattened layer, Bi-directional Long-short Term Memory (Bi-LSTM), and Attention LSTM. This research utilized an effective multimodal dataset named VREED (VR Eyes: Emotions Dataset) for emotion recognition. The dataset was first reduced to binary and multi-class categories. We then processed the dataset to handle missing values and applied normalization techniques to enhance data consistency. Subsequently, explainable Machine Learning (ML) and Deep Learning (DL) classifiers were employed to predict emotions in VR. Experimental analysis and results indicate that the Attention LSTM model excelled in binary classification, while both DNN and Attention LSTM achieved outstanding performance in multi-class classification, with up to 99.99% accuracy. These findings underscore the efficacy of integrating VR with advanced, explainable ML and DL methods for emotion recognition.

USE OF VIDEO-BASED DATA TO INFORM PSYCHOSOCIAL INTERVENTIONS FOR PEOPLE LIVING WITH DEMENTIA

Abstract The pandemic has accelerated technological developments in care services. Emerging evidence suggests feasibility of delivering evidence-based psychosocial interventions such as group cognitive stimulation therapy using information and communication technology (ICT) platforms. A unique opportunity offered by ICT is the more readily available data at session level to investigate nuanced, theoretically based outcome predictors, to improve efficacy of these complex interventions. Theoretically, the cognitive and wellbeing benefits of group cognitive stimulation therapy is related to the level of person-centred engagement. As part of an ongoing randomised controlled trial involving ICT-delivered group cognitive stimulation therapy, this study examines the use of big data analytics of session behavioural data to develop predictive algorithms of cognitive and wellbeing outcomes. In 78 people living with mild to moderate dementia from 26 groups, we create automatically estimated markers of facial expression and eye movement pattern, and hand-coded markers (using time-sampling method) of engagement level and speech/verbal production information, to autobiographical information presented in the session. These markers are used to predict changes in cognitive and wellbeing outcomes. Pilot analysis using hand-coded speech sample suggested strong correlation between time-sampled engagement level with cognitive benefits, and changes in information content and fluency with cognitive benefits after the group intervention (ranging from r=0.76 to 0.97, p< 0.001). The markers we develop in this study using video-based data from ICT-delivered therapy sessions, including deep learning-based emotion recognition and eye movement analysis with hidden Markov model, allow further application in understanding the mechanisms of action in different psychosocial interventions.

Deep Learning-Based Emotion Recognition through Facial Expressions

Nowadays, humans can communicate easily with others by recognizing speech and text characters, particularly facial expressions. In human communication, it is critical to comprehend their emotion or implicit expression. Indeed, facial expression recognition is vital for analyzing the emotions of conversation partners, which can contribute to a series of matters, including mental health consulting. This technique enables psychiatrists to select appropriate questions based on their patients’ current emotional state. The purpose of this study was to develop a deep learningbased model for detecting and recognizing emotions on human faces. We divided the experiment into two parts: Faster R-CNN and mini-Xception architecture. We concentrated on four distinct emotional states: angry, sad, happy, and neutral. Both models implemented using the Faster R-CNN and the mini-Xception architectures were compared during the evaluation process. The findings indicate that the mini-Xception architecture model produced a better result than the Faster R-CNN. This study will be expanded in the future to include the detection of complex emotions such as sadness.

Deep Learning Approach for Emotion Recognition Analysis in Text Streams

Social media sites employ various approaches to track feelings, including diagnosing neurological problems, including fear, in people or assessing a population public sentiment. One essential obstacle for automatic emotion recognition principles is variable with fluctuating limitations, language, and interpretation shifts. Therefore, in this paper, a deep learning-based emotion recognition (DL-EM) system has been proposed to describe the various relational effects in emotional groups. A soft classification method is suggested to quantify the tendency and allocate a message to each emotional class. A supervised framework for emotions in text streaming messages is developed and tested. Two of the major activities are offline teaching assignments and interactive emotion classification techniques. The first challenge offers templates in text responses to describe sentiment. The second activity includes implementing a two-stage framework to identify live broadcasts of text messages for dedicated emotion monitoring.

Emotion recognition while applying cosmetic cream using deep learning from EEG data; cross-subject analysis.

We report a deep learning-based emotion recognition method using EEG data collected while applying cosmetic creams. Four creams with different textures were randomly applied, and they were divided into two classes, "like (positive)" and "dislike (negative)", according to the preference score given by the subject. We extracted frequency features using well-known frequency bands, i.e., alpha, beta and low and high gamma bands, and then we created a matrix including frequency and spatial information of the EEG data. We developed seven CNN-based models: (1) inception-like CNN with four-band merged input, (2) stacked CNN with four-band merged input, (3) stacked CNN with four-band parallel input, and stacked CNN with single-band input of (4) alpha, (5) beta, (6) low gamma, and (7) high gamma. The models were evaluated by the Leave-One-Subject-Out Cross-Validation method. In like/dislike two-class classification, the average accuracies of all subjects were 73.2%, 75.4%, 73.9%, 68.8%, 68.0%, 70.7%, and 69.7%, respectively. We found that the classification performance is higher when using multi-band features than when using single-band feature. This is the first study to apply a CNN-based deep learning method based on EEG data to evaluate preference for cosmetic creams.

Accelerating 3D Convolutional Neural Network with Channel Bottleneck Module for EEG-Based Emotion Recognition.

Deep learning-based emotion recognition using EEG has received increasing attention in recent years. The existing studies on emotion recognition show great variability in their employed methods including the choice of deep learning approaches and the type of input features. Although deep learning models for EEG-based emotion recognition can deliver superior accuracy, it comes at the cost of high computational complexity. Here, we propose a novel 3D convolutional neural network with a channel bottleneck module (CNN-BN) model for EEG-based emotion recognition, with the aim of accelerating the CNN computation without a significant loss in classification accuracy. To this end, we constructed a 3D spatiotemporal representation of EEG signals as the input of our proposed model. Our CNN-BN model extracts spatiotemporal EEG features, which effectively utilize the spatial and temporal information in EEG. We evaluated the performance of the CNN-BN model in the valence and arousal classification tasks. Our proposed CNN-BN model achieved an average accuracy of 99.1% and 99.5% for valence and arousal, respectively, on the DEAP dataset, while significantly reducing the number of parameters by 93.08% and FLOPs by 94.94%. The CNN-BN model with fewer parameters based on 3D EEG spatiotemporal representation outperforms the state-of-the-art models. Our proposed CNN-BN model with a better parameter efficiency has excellent potential for accelerating CNN-based emotion recognition without losing classification performance.

Deep Learning Based Emotion Recognition and Visualization of Figural Representation.

This exploration aims to study the emotion recognition of speech and graphic visualization of expressions of learners under the intelligent learning environment of the Internet. After comparing the performance of several neural network algorithms related to deep learning, an improved convolution neural network-Bi-directional Long Short-Term Memory (CNN-BiLSTM) algorithm is proposed, and a simulation experiment is conducted to verify the performance of this algorithm. The experimental results indicate that the Accuracy of CNN-BiLSTM algorithm reported here reaches 98.75%, which is at least 3.15% higher than that of other algorithms. Besides, the Recall is at least 7.13% higher than that of other algorithms, and the recognition rate is not less than 90%. Evidently, the improved CNN-BiLSTM algorithm can achieve good recognition results, and provide significant experimental reference for research on learners’ emotion recognition and graphic visualization of expressions in an intelligent learning environment.

EEG-Based Estimation on the Reduction of Negative Emotions for Illustrated Surgical Images.

Electroencephalogram (EEG) biosignals are widely used to measure human emotional reactions. The recent progress of deep learning-based classification models has improved the accuracy of emotion recognition in EEG signals. We apply a deep learning-based emotion recognition model from EEG biosignals to prove that illustrated surgical images reduce the negative emotional reactions that the photographic surgical images generate. The strong negative emotional reactions caused by surgical images, which show the internal structure of the human body (including blood, flesh, muscle, fatty tissue, and bone) act as an obstacle in explaining the images to patients or communicating with the images with non-professional people. We claim that the negative emotional reactions generated by illustrated surgical images are less severe than those caused by raw surgical images. To demonstrate the difference in emotional reaction, we produce several illustrated surgical images from photographs and measure the emotional reactions they engender using EEG biosignals; a deep learning-based emotion recognition model is applied to extract emotional reactions. Through this experiment, we show that the negative emotional reactions associated with photographic surgical images are much higher than those caused by illustrated versions of identical images. We further execute a self-assessed user survey to prove that the emotions recognized from EEG signals effectively represent user-annotated emotions.

EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention

Emotion recognition based on electroencephalography (EEG) is a significant task in the brain-computer interface field. Recently, many deep learning-based emotion recognition methods are demonstrated to outperform traditional methods. However, it remains challenging to extract discriminative features for EEG emotion recognition, and most methods ignore useful information in channel and time. This paper proposes an attention-based convolutional recurrent neural network (ACRNN) to extract more discriminative features from EEG signals and improve the accuracy of emotion recognition. First, the proposed ACRNN adopts a channel-wise attention mechanism to adaptively assign the weights of different channels, and a CNN is employed to extract the spatial information of encoded EEG signals. Then, to explore the temporal information of EEG signals, extended self-attention is integrated into an RNN to recode the importance based on intrinsic similarity in EEG signals. We conducted extensive experiments on the DEAP and DREAMER databases. The experimental results demonstrate that the proposed ACRNN outperforms state-of-the-art methods.

Construction of facial emotion database through subjective experiments and its application to deep learning-based facial image processing

As the development of interactive robots and machines, studies to understand and reproduce facial emotions by computers have become important research areas. For achieving this goal, several deep learning-based facial image analysis and synthesis techniques recently have been proposed. However, there are difficulties in the construction of facial image dataset having accurate emotion tags (annotations, metadata), because such emotion tags significantly depend on human perception and cognition. In this study, we constructed facial image dataset having accurate emotion tags through subjective experiments. First, based on image retrieval using the emotion terms, we collected more than 1,600,000 facial images from SNS. Next, based on a face detection image processing, we obtained approximately 380,000 facial region images as “big data.” Then, through subjective experiments, we manually checked the facial expression and the corresponding emotion tags of the facial regions. Finally, we achieved approximately 5,500 facial images having accurate emotion tags as “good data.” For validating our facial image dataset in deep learning-based facial image analysis and synthesis, we applied our dataset to CNN-based facial emotion recognition and GAN-based facial emotion reconstruction. Through these experiments, we confirmed the feasibility of our facial image dataset in deep learning-based emotion recognition and reconstruction.

Facial Data Visualization for Improved Deep Learning Based Emotion Recognition

Facial Data Visualization for Improved Deep Learning Based Emotion Recognition

Towards Emotionally Aware AI Smart Classroom: Current Issues and Directions for Engineering and Education

Future smart classrooms that we envision will significantly enhance learning experience and seamless communication among students and teachers using real-time sensing and machine intelligence. Existing developments in engineering have brought the state-of-the-art to an inflection point, where they can be utilized as components of a smart classroom. In this paper, we propose a smart classroom system that consists of these components. Our proposed system is capable of making real-time suggestions to an in-class presenter to improve the quality and memorability of their presentation by allowing the presenter to make real-time adjustments/corrections to their non-verbal behavior, such as hand gestures, facial expressions, and body language. We base our suggested system components on existing research in affect sensing, deep learning-based emotion recognition, and real-time mobile-cloud computing. We provide a comprehensive study of these technologies and determine the computational requirements of a system that incorporates these technologies. Based on these requirements, we provide a feasibility study of the system. Although the state-of-the-art research in most of the components we propose in our system are advanced enough to realize the system, the main challenge lies in: 1) the integration of these technologies into a holistic system design; 2) their algorithmic adaptation to allow real-time execution; and 3) quantification of valid educational variables for use in algorithms. In this paper, we discuss current issues and provide future directions in engineering and education disciplines to deploy the proposed system.