Emotion Recognition Results Research Articles

Research on Emotional Recognition System of EEG Signals Based on CNN+LSTM

We studied an emotion recognition system based on EEG signals, conducted MATLAB simulation for EEG signal collection, and conducted experiments on EEG signal collection using STM32 microcontroller. We proposed an EEG emotion recognition system based on convolutional neural network (CNN) and short-term memory artificial neural network (LSTM) models. The model used gradient descent algorithm and cross entropy loss function algorithm, and was used the DEAP dataset to verify the accuracy of emotion recognition results, which reached 94.023%, The accuracy of emotion recognition results which is achieved by running the Python trained model on Raspberry Pi reached 98.98%. After compared the data collection information and tested the software and hardware, the experimental results showed that the EEG signal collection system can achieve the functions of EEG signal data collection and improve emotion recognition rate.

MULTIMODAL SYSTEM FOR FACIAL EMOTION RECOGNITION BASED ON DEEP LEARNING

Emotions are one of the main ways of communication between people and of expressing attitudes towards objects, products, services, etc. They are divided to verbal and non-verbal classes. Human speech and intonation belong to the first class, and to the second (non-verbal) facial and body emotions, known as body language. The subject of this report is the development of multimodal deep learning system intended to recognize facial and body emotions and their relationship with the scene (weather) in which they occur. It is based on three deep learning neural networks (DNN) each one for recognition of facial emotion, body emotion and weather. Combining their results, we improve significantly the final facial emotion recognition (FER) results.

MVIB-DVA: Learning minimum sufficient multi-feature speech emotion embeddings under dual-view aware

Speech emotion recognition (SER) is a crucial topic in human–computer interaction. However, there are still many challenges to extracting emotional embeddings. Emotional embeddings extracted by network models often contain noise and incomplete emotional information. To meet these challenges, this study developed an innovative model (MVIB-DVA) composed of a multi-feature variational information bottleneck (MVIB) based on the information bottleneck (IB) principle and a dual-view aware module (DVAM) with an attention mechanism. MVIB employs the IB principle as the driving model and utilizes learned minimal sufficient single-feature emotional embeddings as auxiliary information. The aims are to capture unique emotional information in individual features and complementary information between different types of features as well as reduce noise and represent rich emotional information with fewer parameters. DVAM proposes (1) a frequency-domain statistical aware module (FDSAM) in the frequency domain that emphasizes the frequency that best reflects emotional information and (2) a frame aware module (FAM) in the time domain that focuses on the frames that contribute to the most to the final emotion recognition results. A separate channel extracts details ignored in the frequency and time domain views, extracting more comprehensive emotional information. The experimental results show that our method performs excellently in recognizing speech emotions. MVIB-DVA achieved weighted accuracy (WA) of 74.05% and unweighted accuracy (UA) of 75.67% on the IEMOCAP dataset. Similarly, on the RAVDESS dataset, MVIB-DVA attained WA of 86.66% and UA of 86.51%.

Constructing a Multimodal Music Teaching Model in College by Integrating Emotions

Abstract In this study, we enhanced the CaffeNet network for recognizing students’ facial expressions in a music classroom and extracted emotional features from their expressions. Additionally, students’ speech signals were processed through filters to identify emotional characteristics. Using the LRLR fusion strategy, these expression and speech-based emotional features were combined to derive multimodal fusion emotion results. Subsequently, a music teaching model incorporating this multimodal emotion recognition was developed. Our analysis indicates a mere 6.03% discrepancy between the model’s emotion recognition results and manual emotional assessments, underscoring its effectiveness. Implementation of this model in a music teaching context led to a noticeable increase in positive emotional responses—happy and surprised emotions peaked at 30.04% and 27.36%, respectively, during the fourth week. Furthermore, 70% of students displayed a positive learning status, demonstrating a significant boost in engagement and motivation for music learning. This approach markedly enhances student interest in learning and provides a solid basis for improving educational outcomes in music classes.

Emotion Analysis Using Improved Cat Swarm Optimization with Machine Learning for Speech-impaired People

Emotion analysis is an effective method for improving interaction and understanding for speech-impaired people. We can provide the best interaction and support emotional well-being by analyzing the emotion conveyed through speech. Using deep learning (DL) or machine learning algorithms for training an emotion classification method. This might include training classifiers namely random forests, deep neural networks, or support vector machines. It is noteworthy that emotion analysis could be effective; however, it is crucial to consider individual differences and context while interpreting emotion. Furthermore, ensuring data protection and privacy and obtaining consent are vital features to consider while working with sensitive speech data. Therefore, this study presents an emotion analysis approach using improved cat swarm optimization with machine learning (EA-ICSOML) technique. The EA-ICSOML technique applies the concepts of computer vision and DL to identify various types of emotions. For feature vector generation, the ShuffleNet model is used in this work. To adjust the hyperparameters compared to the ShuffleNet system, the ICSO algorithm is used. Finally, the recognition and classification of emotions are performed using the Transient Chaotic Neural Network approach. The performance validation of the EA-ICSOML technique is validated on facial emotion databases. The simulation result inferred the improved emotion recognition results of the EA-ICSOML approach compared to other recent models in terms of different evaluation measures.

Frame-Level Teacher-Student Learning With Data Privacy for EEG Emotion Recognition.

Recently, electroencephalogram (EEG) emotion recognition has gradually attracted a lot of attention. This brief designs a novel frame-level teacher-student framework with data privacy (FLTSDP) for EEG emotion recognition. The framework first proposes a teacher-student network without prior professional information for automated filtering of useful frame-level features by a gated mechanism and extracting high-level features by using knowledge distillation to capture the results of EEG emotion recognition from a teacher network and student networks. Then, the results from subnetworks are integrated by using the novel decision module, which, motivated by the voting mechanism, adjusts the composition of feature vectors and improves the weight of accurate prediction to optimize the integration effect. During training, an innovative data privacy protection mechanism is applied for avoiding data sharing, where each student network only inherits weights from all trained networks and does not inherit the training dataset. Here, the framework can be repeatedly optimized and improved by only training the next student subnetwork on new EEG signals. Experimental results show that our framework improves the accuracy of EEG emotion recognition by more than 5% and gets state-of-the-art performance for EEG emotion recognition in the subject-independent mode.

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals.

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

A standardized database of Chinese emotional short videos based on age and gender differences.

Most of the existing emotion elicitation databases use the film clips as stimuli and do not take into account the age and gender differences of participants. Considering the short videos have the advantages of short in time, easy to understand and strong emotional appeal, we choose them to construct a standardized database of Chinese emotional short videos by the joint analysis of age and gender differences. Two experiments are performed to establish and validate our database. In the Experiment 1, we selected 240 stimuli from 2700 short videos and analyzed the subjective evaluation results of 360 participants with different ages and genders. As a result, a total of 54 short videos with three categories of emotions were picked out for 6 groups of participants, including the male and female respectively aged in 20-24, 25-29 and 30-34. In the Experiment 2, we recorded the EEG signals and subjective experience scores of 81 participants while watching different video stimuli. Both the results of EEG emotion recognition and subjective evaluation indicate that our database of 54 short videos can achieve better emotion elicitation effects compared with film clips. Furthermore, the targeted delivery of specific short videos has also been verified to be effective, helping the researchers choose appropriate emotional elicitation stimuli for different participants and promoting the study of individual differences in emotion responses.

Sentiment recognition model of EEG signals combined with one-dimensional convolution and BiBASRU-AT

To address the problems of low accuracy of EEG emotion sentiment and insufficient feature extraction ability of recurrent model, an EEG sentiment recognition model combining one-dimensional convolution and BiBASRU-AT is proposed. The data set is preprocessed in segments to expand the number of samples, and 62 channel local emotional features are extracted from one-dimensional convolution; The built-in self-attention simple recurrent unit is constructed to capture the multi-channel fusion features and the dependence between channels. The soft attention mechanism identifies the key features that have a great impact on the identification of emotional tendencies, and the linear layer outputs the positive, neutral and negative emotion recognition results. The experimental results on the EEG data set(SEED) show that the model achieves an average classification accuracy of 90.24%, which is higher than the excellent deep learning model compared with the experiment. The built-in self-attention simple recurrent unit has stronger feature capture ability, which proves the effectiveness of the model.

A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles: Emotions Triggered by Video-Audio Clips in Driving Scenarios

In this article, a new dataset, the driver emotion facial expression (DEFE) dataset for drivers’ spontaneous emotions analysis is introduced. The dataset includes facial expression recordings from 60 participants during driving. After watching a selected video-audio clip to elicit a specific emotion, each participant completed the driving tasks in the same driving scenario and rated his/her emotional responses during the driving processes from the aspects of dimensional emotion method and discrete emotion method. The study also conducted classification experiments to recognize the scales of arousal, valence, dominance, as well as the emotion category and intensity to establish baseline results for the proposed dataset. Furthermore, this paper compared emotion recognition results difference through facial expressions between dynamic driving and static life scenarios. The results showed that dynamic driving and static life datasets were different in emotion recognition results. To further explore the reasons for the difference in emotion recognition results, the analysis from the AU (action unit) presence perspective was studied. The results showed significant differences in the AUs presence of facial expressions between dynamic driving and static life scenarios, indicating that drivers’ facial expressions may be affected by the driving task to influence the recognition of drivers’ emotions through facial expressions. Therefore, to accurately recognize the drivers’ emotions to establish a reliable emotion-aware human-machine interaction system, thereby improving driving safety and comfort, publishing a human emotion dataset specifically for the driver is necessary. The proposed dataset will be publicly available so that researchers worldwide can use it to develop and examine their driver emotion analysis methods. To the best of our knowledge, this is currently the only public driver facial expression dataset.

Exploration of effective electroencephalography features for the recognition of different valence emotions.

Recent studies have shown that the recognition and monitoring of different valence emotions can effectively avoid the occurrence of human errors due to the decline in cognitive ability. The quality of features directly affects emotion recognition results, so this manuscript explores the effective electroencephalography (EEG) features for the recognition of different valence emotions. First, 110 EEG features were extracted from the time domain, frequency domain, time-frequency domain, spatial domain, and brain network, including all the current mainly used features. Then, the classification performance, computing time, and important electrodes of each feature were systematically compared and analyzed on the self-built dataset involving 40 subjects and the public dataset DEAP. The experimental results show that the first-order difference, second-order difference, high-frequency power, and high-frequency differential entropy features perform better in the recognition of different valence emotions. Also, the time-domain features, especially the first-order difference features and second-order difference features, have less computing time, so they are suitable for real-time emotion recognition applications. Besides, the features extracted from the frontal, temporal, and occipital lobes are more effective than others for the recognition of different valence emotions. Especially, when the number of electrodes is reduced by 3/4, the classification accuracy of using features from 16 electrodes located in these brain regions is 91.8%, which is only about 2% lower than that of using all electrodes. The study results can provide an important reference for feature extraction and selection in emotion recognition based on EEG.

Learning multi-scale features for speech emotion recognition with connection attention mechanism

Speech emotion recognition (SER) has become a crucial topic in the field of human–computer interactions. Feature representation plays an important role in SER, but there are still many challenges in feature representation such as the inability to predict which features are most effective for SER and the cultural differences in emotion expression. Most previous studies use a single type of feature for the recognition task or conduct early fusion of features. However, a single type of feature cannot well reflect the emotions of speech signals. Also, different features contain different information, direct fusion cannot integrate the advantages of different features. To overcome these challenges, this paper proposes a parallel network for multi-scale SER based on a connection attention mechanism (AMSNet). AMSNet fuses fine-grained frame-level manual features with coarse-grained utterance-level deep features. Meanwhile, it adopts different speech emotion feature extraction modules according to the temporal and spatial features of speech signals, which enriches features and improves feature characterization. The network consists of a frame-level representation learning module (FRLM) based on the time structure and an utterance-level representation learning module (URLM) based on the global structure. Besides, improved attention-based long short-term memory (LSTM) is introduced into FRLM to focus on the frames that contribute more to the final emotion recognition result. In URLM, a convolutional neural network with the squeeze-and-excitation block (SCNN) is introduced to extract deep features. In addition, the connection attention mechanism is proposed for feature fusion, which applies different weights to different features. Extensive experiments are conducted on the IEMOCAP and EmoDB datasets, and the results demonstrate the effectiveness and performance superiority of AMSNet. Our code will be publicly available at https://codeocean.com/capsule/8636967/tree/v1.

Troi

Emotional Self-Awareness (ESA) plays a vital role in physical and mental well-being. Recent advancements in artificial intelligence technologies have shown promising emotion recognition results, opening new opportunities to build systems to support ESA. However, little research has been done to understand users' perspectives on artificial-intelligence-based emotion recognition systems. We introduce Troi, an automatic emotion recognition mobile app using wearable signals. With Troi, we ran a multi-day user study with 12 users to understand user preference parameters, such as perceived accuracy, confidence, preferred emotion representations, effect of self-awareness of emotions, and real-time use cases. Further, we extend our study to evaluate the machine learning model in-the-wild to understand behaviours in-the-wild. We found that users perceived accuracy of the emotion recognition model is higher than the actual model prediction accuracy; there was no strong preference for one specific emotion representation, and users' self-awareness of emotions improved over time.

Design of Blended Teaching Model Based on Emotion Recognition and Language Learning.

With the vigorous development of Internet technology, great changes have taken place in all aspects of human society. This change is also having an increasingly significant impact on the education sector. It is even a trend to subvert the tradition. This also makes the student's identity a passive recipient of knowledge. At the same time, the final orientation of China's education model is the result of examinations, and there is little guidance for students' interest. Of course, traditional teaching also has the ability to enable students' subject knowledge to be systematically established, and to communicate with teachers in the teaching process, which can improve students' learning efficiency in the classroom. In the face of the information explosion in today's society and the rapid development of the above-mentioned technologies, the change of students' learning mode of knowledge has also ushered in an opportunity to change. The purpose of this paper is to study the establishment of a blended teaching model that combines traditional classrooms with network applications. With the help of the characteristics of network big data, it transforms students' passive learning identities, and combines offline traditional learning classrooms with online learning. In addition to the advantages mentioned above, the advantage of online learning is that some network science and technology can be used for the online learning platform to serve the entire teaching process. Therefore, this paper proposes a blended teaching model based on the network platform for students' emotion recognition and language learning result analysis. And from the experimental results in this paper, it can be seen that the recognition evaluation rate of HTMC, the feature of emotion recognition, is 71.52% and the recognition frequency of ETMC is 73.89%. The above two recognition parameters can better reflect the emotional changes in the mixed teaching process.

Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion.

With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis.

Algorithm Composition and Emotion Recognition Based on Machine Learning.

This paper proposes a new algorithm composition network from the perspective of machine learning, based on an in-depth study of related literature. At the same time, this paper examines the characteristics of music and develops a model for recognising musical emotions. Using the model's information entropy of pitch and intensity to extract the main melody track, note features are extracted from bar features. Finally, the cosine of the vector included angle is used to judge the similarity between feature vectors of several adjacent sections, allowing the music to be divided into several independent segments. The emotional model of music is used to analyze each segment's emotion. By quantifying music features, this paper classifies and quantifies music emotion based on the mapping relationship between music features and emotion. Music emotion can be accurately identified by the model. The model's emotion recognition accuracy is up to 93.78 percent, and the algorithm's recall rate is up to 96.3 percent, according to simulation results. The recognition method used in this paper has a higher recognition ability than other methods, and the emotion recognition result is more reliable. This paper can not only meet the composer's auxiliary creative needs, but it can also help intelligent music services.

Graph-Based Object Semantic Refinement for Visual Emotion Recognition

The rich semantic information contained in images is an important clue to explore visual emotions. Therefore, exploring the correlation between visual emotion and the semantic relationship of objects, and extracting more effective semantic features through explicit or implicit modeling is very important for visual emotion analysis. In this paper, a novel Graph-based Object Semantic Refinement (GOSR) model is proposed to extract multi-level semantic features for visual emotion classification, in which graph structures is used to represent the object semantics and their position relationships of an image, and Graph Convolutional Networks (GCN) is used to refine object information by the aggregating neighbor object with their position relationships. The different convolutional layer’s features from GCN are further fused by Gated Recurrent Units (GRU) networks to achieve high-level semantic features. Then a framework with two branches to leverage visual and semantic information for visual sentiment analysis is proposed, which uses convolutional neural networks to extract visual features from images, and collaborates with semantic features from GOSR model to achieve better emotion recognition results. Besides, for alleviating the potentially unreasonable predictions and promote models collaboration, a novel tendency loss function based on the correlations among emotion labels is proposed to adjust the output activation value other than the target label. Extensive experiments on four widely used benchmark datasets show that our proposed method can achieve competitive performance and outperform most of the state-of-the-art methods on visual emotion recognition.

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

Recently, emotion recognition in real sceneries such as in the wild has attracted extensive attention in affective computing, because existing spontaneous emotions in real sceneries are more challenging and difficult to identify than other emotions. Motivated by the diverse effects of different lengths of audio spectrograms on emotion identification, this paper proposes a multiscale deep convolutional long short-term memory (LSTM) framework for spontaneous speech emotion recognition. Initially, a deep convolutional neural network (CNN) model is used to learn deep segment-level features on the basis of the created image-like three channels of spectrograms. Then, a deep LSTM model is adopted on the basis of the learned segment-level CNN features to capture the temporal dependency among all divided segments in an utterance for utterance-level emotion recognition. Finally, different emotion recognition results, obtained by combining CNN with LSTM at multiple lengths of segment-level spectrograms, are integrated by using a score-level fusion strategy. Experimental results on two challenging spontaneous emotional datasets, i.e., the AFEW5.0 and BAUM-1s databases, demonstrate the promising performance of the proposed method, outperforming state-of-the-art methods.

Prediction of Sports Aggression Behavior and Analysis of Sports Intervention Based on Swarm Intelligence Model

In the process of sports, athletes often have aggressive behaviors because of their emotional fluctuations. This violent sports behavior has caused many serious bad effects. In order to reduce and solve this kind of public emergencies, this paper aims to create a swarm intelligence model for predicting people's sports attack behavior, takes the swarm intelligence algorithm as the core technology optimization model, and uses the Internet of Things and other technologies to recognize emotions on physiological signals, predict, and intervene sports attack behavior. The results show the following: (1) After the 50-fold cross-validation method, the results of emotion recognition are good, and the accuracy is high. Compared with other physiological electrical signals, EDA has the worst classification performance. (2) The recognition accuracy of the two methods using multimodal fusion is improved greatly, and the result after comparison is obviously better than that of single mode. (3) Anxiety, anger, surprise, and sadness are the most detected emotions in the model, and the recognition accuracy is higher than 80%. Sports intervention should be carried out in time to calm athletes' emotions. After the experiment, our model runs successfully and performs well, which can be optimized and tested in the next step.

Emotion Recognition From EEG Signals of Hearing-Impaired People Using Stacking Ensemble Learning Framework Based on a Novel Brain Network

Emotion recognition based on electroencephalography (EEG) signals has become an interesting research topic in the field of neuroscience, psychology, neural engineering, and computer science. However, the existing studies are mainly focused on normal or depression subjects, and few reports on hearing-impaired subjects. In this work, we have collected the EEG signals of 15 hearing-impaired subjects for categorizing three types of emotions (positive, neutral, and negative). To study the differences in functional connectivity between normal and hearing-impaired subjects under different emotional states, a novel brain network stacking ensemble learning framework was proposed. The phase-locking value (PLV) was utilized to calculate the correlation between EEG channels, and then we constructed a brain network using double thresholds. The spatial features of the brain network were extracted from the perspectives of local differentiation and global integration. In addition, the stacking ensemble learning framework was used to classify the fused features. To evaluate the proposed model, extensive experiments were carried out on the SEED dataset, and the result shows that the proposed method achieved superior performance than state-of-the-art models, in which the average classification accuracy is 0.955 (std: 0.052). In addition, the experimental results of hearing-impaired emotion recognition show that the average classification accuracy is 0.984 (std: 0.005). Finally, we investigated the activation patterns to reveal important brain regions and inter-channel relations about emotion recognition.