Emotion Recognition Results Research Articles

Research on emotion recognition method of weightlifters based on a non-negative matrix decomposition algorithm

In order to overcome the problems of high recognition error rate and low recognition efficiency existing in the traditional method of emotional recognition of weightlifters, a method of emotional recognition of weightlifters based on a non-negative matrix decomposition algorithm is proposed. The database of emotion recognition matching criteria is established, and the mapping relationship among emotion, facial expression of far mobilisation and weight lifting posture is formed. The real-time motion images of weightlifters are collected, and the image classifier is designed by using the non-negative matrix decomposition algorithm. The real-time images are divided into facial expression images and whole body motion images, and the features of the two images are extracted respectively. The extracted results are matched with the results of the standard database, and the real-time emotion recognition results of weightlifters are obtained. The experimental results show that compared with the traditional emotion recognition method, the proposed method improves the accuracy of emotion recognition by 50%.

Emotion Analysis of College Students Using a Fuzzy Support Vector Machine

With the rapid development of society, the number of college students in our country is on the rise. College students are under pressure due to challenges from the society, school, and family, but they cannot find a suitable solution. As a result, the psychological problems of college students are diversified and complicated. The mental health problem of college students is becoming more and more serious, which requires urgent attention. This article realizes the monitoring of university mental health by identifying and analyzing the emotions of college students. This article uses EEG to determine the emotional state of college students. First, feature extraction is performed on different rhythm data of EEG, and then a fuzzy support vector machine (FSVM) is used for classification. Finally, a decision fusion mechanism based on the D-S evidence combination theory is used to fuse the classification results and output the final emotion recognition results. The contribution of this research is mainly in three aspects. One is the use of multiple features, which improves the efficiency of data use; the other is the use of a fuzzy support vector machine classifier with higher noise resistance, and the recognition rate of the model is better. The third is that the decision fusion mechanism based on the D-S evidence combination theory takes into account the classification results of each feature, and the classification results assist each other and integrate organically. The experiment compares emotion recognition based on single rhythm, multirhythm combination, and multirhythm fusion. The experimental results fully prove that the proposed emotion recognition method can effectively improve the recognition efficiency. It has a good practical value in the emotion recognition of college students.

Speech Emotion Recognition Based on Selective Interpolation Synthetic Minority Over-Sampling Technique in Small Sample Environment.

Speech emotion recognition often encounters the problems of data imbalance and redundant features in different application scenarios. Researchers usually design different recognition models for different sample conditions. In this study, a speech emotion recognition model for a small sample environment is proposed. A data imbalance processing method based on selective interpolation synthetic minority over-sampling technique (SISMOTE) is proposed to reduce the impact of sample imbalance on emotion recognition results. In addition, feature selection method based on variance analysis and gradient boosting decision tree (GBDT) is introduced, which can exclude the redundant features that possess poor emotional representation. Results of experiments of speech emotion recognition on three databases (i.e., CASIA, Emo-DB, SAVEE) show that our method obtains average recognition accuracy of 90.28% (CASIA), 75.00% (SAVEE) and 85.82% (Emo-DB) for speaker-dependent speech emotion recognition which is superior to some state-of-the-arts works.

Multimodal Fused Emotion Recognition About Expression-EEG Interaction and Collaboration Using Deep Learning

The proposed emotion recognition model is based on the hierarchical long-short term memory neural network (LSTM) for video-electroencephalogram (Video-EEG) signal interaction. The inputs are facial-video and EEG signals from the subjects when they are watching the emotion-stimulated video. The outputs are the corresponding emotion recognition results. Facial-video features and corresponding EEG features are extracted based on a fully connected neural network (FC) at each time point. These features are fused through hierarchical LSTM to predict the key emotional signal frames at the next time point until the emotion recognition result is calculated at the last time point. Specially, a self-attention mechanism is applied to show the correlation of the stacked LSTM at different hierarchies. In this process, the “selective focus” is used to analyze the human-emotional temporal sequences in each model, which improves the utilization of the key spatial EEG signals. Moreover, the process includes the temporal attention mechanism to predict the key signal frame at next time point, which utilizes the key emotion data in temporal domain. The experimental results prove that the classification rate (CR) and F1-score of the proposed emotion recognition model are significantly increased by at least 2% and 0.015, respectively, compared to other methods.

CreativeBioMan: A Brain- and Body-Wearable, Computing-Based, Creative Gaming System

Current artificial intelligence (AI) technology is mainly used in rational work such as computation and logical analysis. How to make the machine as aesthetic and creative as humans has gradually gained attention. This paper presents a creative game system (i.e., CreativeBioMan) for the first time. It combines brain wave data and multimodal emotion data, and then uses an AI algorithm for intelligent decision fusion, which can be used in artistic creation, aiming at separating the artist from repeated labor creation. To imitate the process of humans' artistic creation, the creation process of the algorithm is related to artists' previous artworks and their emotion. EEG data is used to analyze the style of artists and then match them with a style from a data set of historical works. Then, universal AI algorithms are combined with the unique creativity of each artist that evolve into a personalized creation algorithm. According to the results of cloud emotion recognition, the color of the artworks is corrected so that the artist's emotions are fully reflected in the works, and thus novel works of art are created. This allows the machine to integrate the understanding of past art and emotions with the ability to create new art forms, in the same manner as humans. This paper introduces the system architecture of CreativeBioMan from two aspects: data collection of the brain and body wearable devices, as well as the intelligent decision-making fusion of models. A Testbed platform is built for an experiment and the creativity of the works produced by the system is analyzed.

Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging

The immanent dependencies between audio and visual modalities extracted from video content and the well-established film grammar (i.e., domain knowledge) are important for emotion video recognition and regression. However, these tools have yet to be exploited successfully. Therefore, we propose a multimodal deep regression Bayesian network (MMDRBN) to capture the relationship between audio and visual modalities for emotion video tagging. We then modify the structure of the MMDRBN to incorporate domain knowledge. A regression Bayesian network (RBN) is formed from one latent layer, one visible layer and directed links from the latent layer to the visible layer. RBN is able to fully represent the data, since it captures the dependencies not only among the visible variables but also among the latent variables given visible variables. For the MMDRBN, first, we learn several layers of RBNs using audio and visual modalities, and then stack these RBNs to form two deep networks. A joint representation is obtained from the top layers of the two deep networks, capturing the deep dependencies between audio and visual modalities. We also summarize the main audio and visual elements used by filmmakers to convey emotions and formulate them as semantical meaningful middle-level representation, i.e., attributes. Through these attributes, we construct the knowledge-augmented MMDRBN, which learns a hybrid middle-level video representation using video data and the summarized attributes. Experimental results of both emotion recognition and regression from videos on the LIRIS-ACCEDE database demonstrate that the proposed model can successfully capture the intrinsic connections between audio and visual modalities, and integrate the middle-level representation learning from video data and semantical attributes summarized from film grammar. Thus, it achieves superior performance on emotion video tagging compared to state-of-the-art methods.

Further Evidence of the Zero-Association Between Symptoms of Insomnia and Facial Emotion Recognition-Results From a Sample of Adults in Their Late 30s.

Background: Restoring sleep is associated with favorable cognitive, emotional, and behavioral adaptations. As regards the association between sleep duration and facial emotion recognition (FER), results are conflicting, and as regards the association between symptoms of insomnia and FER, no study has been performed so far. Accordingly, the aim of the present study was to investigate whether subjective sleep was associated with FER, along with perceived stress and mental toughness.Method: A total of 201 police officers (mean age = 38.5 years, 64.2% males) took part in the present cross-sectional study. They completed questionnaires covering socio-demographic data, subjective symptoms of insomnia, perceived stress, and mental toughness. Further, they underwent a computerized FER test, consisting of facial emotion labeling and facial emotion matching.Results: Performance of FER (accuracy, speed) was unrelated to subjective symptoms of insomnia. Lower FER was associated with higher age, but not to perceived stress or mental toughness. No gender differences were observed. Higher symptoms of insomnia were associated with higher stress scores and lower scores of mental toughness.Conclusions: The pattern of results suggests that FER was not associated with symptoms of insomnia, understood as a proxy of sleep quality, among adults. This observation replicates those studies showing a zero-association between sleep and FER.

Interpretable Emotion Recognition Using EEG Signals

Electroencephalogram (EEG) signal-based emotion recognition has attracted wide interests in recent years and has been broadly adopted in medical, affective computing, and other relevant fields. However, the majority of the research reported in this field tends to focus on the accuracy of classification whilst neglecting the interpretability of emotion progression. In this paper, we propose a new interpretable emotion recognition approach with the activation mechanism by using machine learning and EEG signals. This paper innovatively proposes the emotional activation curve to demonstrate the activation process of emotions. The algorithm first extracts features from EEG signals and classifies emotions using machine learning techniques, in which different parts of a trial are used to train the proposed model and assess its impact on emotion recognition results. Second, novel activation curves of emotions are constructed based on the classification results, and two emotion coefficients, i.e., the correlation coefficients and entropy coefficients. The activation curve can not only classify emotions but also reveals to a certain extent the emotional activation mechanism. Finally, a weight coefficient is obtained from the two coefficients to improve the accuracy of emotion recognition. To validate the proposed method, experiments have been carried out on the DEAP and SEED dataset. The results support the point that emotions are progressively activated throughout the experiment, and the weighting coefficients based on the correlation coefficient and the entropy coefficient can effectively improve the EEG-based emotion recognition accuracy.

Scoreboard Architectural Pattern and Integration of Emotion Recognition Results

This paper proposes a new design pattern, named Scoreboard , dedicated for applications solving complex, multi-stage, non-deterministic problems. The pattern provides a computational framework for the design and implementation of systems that integrate a large number of diverse specialized modules that may vary in accuracy, solution level, and modality. The Scoreboard is an extension of Blackboard design pattern and comes under behavioral type. The pattern allows for an integration of multimodal results, employing early, and/or late fusion paradigms. Additionally, it provides a framework for the evaluation of the modules, dealing with inconsistency and low accuracy. In this paper, the Scoreboard design pattern is described with the standard meta-data model, followed by a sample implementation. This paper also provides the evaluation results based on experiments and a case study. The evaluation results confirmed the robustness, modularization, ease of integration, efficiency, and adaptability of the solutions with the Scoreboard pattern in comparison with the Blackboard pattern and “no pattern” condition. This paper provides also a case study of Scoreboard application in an integration of emotion recognition results. There are certain complex problems in modern software engineering which require multi-stage, multi-party, multi-modal solutions, and non-deterministic control strategies. Among those are natural language processing, image processing, and emotion recognition, to name just a few. The proposed Scoreboard pattern might be used in the software addressing the problems, especially in research systems that explore large solution spaces and require runtime decisions on execution order.

Towards New Mappings between Emotion Representation Models

There are several models for representing emotions in affect-aware applications, and available emotion recognition solutions provide results using diverse emotion models. As multimodal fusion is beneficial in terms of both accuracy and reliability of emotion recognition, one of the challenges is mapping between the models of affect representation. This paper addresses this issue by: proposing a procedure to elaborate new mappings, recommending a set of metrics for evaluation of the mapping accuracy, and delivering new mapping matrices for estimating the dimensions of a Pleasure-Arousal-Dominance model from Ekman’s six basic emotions. The results are based on an analysis using three datasets that were constructed based on affect-annotated lexicons. The new mappings were obtained with linear regression learning methods. The proposed mappings showed better results on the datasets in comparison with the state-of-the-art matrix. The procedure, as well as the proposed metrics, might be used, not only in evaluation of the mappings between representation models, but also in comparison of emotion recognition and annotation results. Moreover, the datasets are published along with the paper and new mappings might be created and evaluated using the proposed methods. The study results might be interesting for both researchers and developers, who aim to extend their software solutions with affect recognition techniques.

Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality.

Extensive possibilities of applications have made emotion recognition ineluctable and challenging in the field of computer science. The use of non-verbal cues such as gestures, body movement, and facial expressions convey the feeling and the feedback to the user. This discipline of Human–Computer Interaction places reliance on the algorithmic robustness and the sensitivity of the sensor to ameliorate the recognition. Sensors play a significant role in accurate detection by providing a very high-quality input, hence increasing the efficiency and the reliability of the system. Automatic recognition of human emotions would help in teaching social intelligence in the machines. This paper presents a brief study of the various approaches and the techniques of emotion recognition. The survey covers a succinct review of the databases that are considered as data sets for algorithms detecting the emotions by facial expressions. Later, mixed reality device Microsoft HoloLens (MHL) is introduced for observing emotion recognition in Augmented Reality (AR). A brief introduction of its sensors, their application in emotion recognition and some preliminary results of emotion recognition using MHL are presented. The paper then concludes by comparing results of emotion recognition by the MHL and a regular webcam.

Exploring latent semantic information for textual emotion recognition in blog articles

Understanding people's emotions through natural language is a challenging task for intelligent systems based on Internet of Things (IoT). The major difficulty is caused by the lack of basic knowledge in emotion expressions with respect to a variety of real world contexts. In this paper, we propose a Bayesian inference method to explore the latent semantic dimensions as contextual information in natural language and to learn the knowledge of emotion expressions based on these semantic dimensions. Our method synchronously infers the latent semantic dimensions as topics in words and predicts the emotion labels in both word-level and document-level texts. The Bayesian inference results enable us to visualize the connection between words and emotions with respect to different semantic dimensions. And by further incorporating a corpus-level hierarchy in the document emotion distribution assumption, we could balance the document emotion recognition results and achieve even better word and document emotion predictions. Our experiment of the wordlevel and the document-level emotion predictions, based on a well-developed Chinese emotion corpus Ren-CECps, renders both higher accuracy and better robustness in the word-level and the document-level emotion predictions compared to the state-of-theart emotion prediction algorithms.

Analysis of physiological for emotion recognition with the IRS model

Facial expression-based emotion recognition has attracted lots of attention. Higher accurate performance could be expected with help of the other cues, e.g. physiological signals, for some specific blue such as ‘Poker Face’ and the lack of facial expression. In this paper, physiological signals are utilized for inferring user׳s emotion. However, the results of physiological-based emotion recognition are still inaccurate in the user-independent scenario since most existing methods ignore the difference in individual response pattern. To this end, we propose a Group-Based IRS (Individual Response Specificity) model to improve performance of physiological-based emotion recognition by taking user׳s IRS into account. The main contributions of this paper are two-fold: (1) an affective physiological database is collected to analyze human׳s emotional response pattern. The physiological signals are recorded from 30 subjects in four induced emotions (neutral, sadness, fear and pleasure). Three-channel bio-sensors are used to measure users electrocardiogram (ECG), galvanic skin response (GSR) and photo plethysmography (PPG). (2) In the experiment, the Group-based IRS model is proposed for emotion recognition in user-independent scenario, the effectiveness of which has been validated on our database. The results show that the Group-based IRS model can achieve higher recognition precision than the general model.

Weighted spectral features based on local Hu moments for speech emotion recognition

Features greatly influence the results of speech emotion recognition, among which Mel-frequency Cepstral Coefficients (MFCC) is the most commonly used in speech emotion. However, MFCC does not consider both the relationship among neighbor coefficients of Mel filters of a frame and the relationship among coefficients of Mel filters of neighbor frames, which possibly leads to lose many useful features from spectrogram. This paper presents novel weighted spectral features based on Local Hu moments. The idea is motivated by that the energy on spectrogram would drastically vary with some emotion types such as angry and happy, while it would slightly change with other emotion types such as sadness and fear. This phenomenon would affect the local energy distribution of spectrogram in both time axis and frequency axis of spectrogram. To describe local energy distribution of spectrogram, Hu moments computed from local regions of spectrogram are used, as Hu moments can evaluate the degree how the energy is concentrated to the center of energy gravity of local region of spectrogram and can significantly vary with the speech emotion types. The conducted experiments validate the proposed features in terms of the effectiveness of the speech emotion recognition.

Textual emotion recognition for enhancing enterprise computing

The growing interest in affective computing (AC) brings a lot of valuable research topics that can meet different application demands in enterprise systems. The present study explores a sub area of AC techniques – textual emotion recognition for enhancing enterprise computing. Multi-label emotion recognition in text is able to provide a more comprehensive understanding of emotions than single label emotion recognition. A representation of ‘emotion state in text’ is proposed to encompass the multidimensional emotions in text. It ensures the description in a formal way of the configurations of basic emotions as well as of the relations between them. Our method allows recognition of the emotions for the words bear indirect emotions, emotion ambiguity and multiple emotions. We further investigate the effect of word order for emotional expression by comparing the performances of bag-of-words model and sequence model for multi-label sentence emotion recognition. The experiments show that the classification results under sequence model are better than under bag-of-words model. And homogeneous Markov model showed promising results of multi-label sentence emotion recognition. This emotion recognition system is able to provide a convenient way to acquire valuable emotion information and to improve enterprise competitive ability in many aspects.

Exploiting Psychological Factors for Interaction Style Recognition in Spoken Conversation

Determining how a speaker is engaged in a conversation is crucial for achieving harmonious interaction between computers and humans. In this study, a fusion approach was developed based on psychological factors to recognize Interaction Style ($IS$ ) in spoken conversation, which plays a key role in creating natural dialogue agents. The proposed Fused Cross-Correlation Model (FCCM) provides a unified probabilistic framework to model the relationships among the psychological factors of emotion, personality trait ($PT$), transient $IS$, and $IS$ history, for recognizing $IS$. An emotional arousal-dependent speech recognizer was used to obtain the recognized spoken text for extracting linguistic features to estimate transient $IS$ likelihood and recognize $PT$. A temporal course modeling approach and an emotional sub-state language model, based on the temporal phases of an emotional expression, were employed to obtain a better emotion recognition result. The experimental results indicate that the proposed FCCM yields satisfactory results in $IS$ recognition and also demonstrate that combining psychological factors effectively improves $IS$ recognition accuracy.

FOOD PROCESSOR WITH RECOGNITION ABILITY OF EMOTION-RELATED INFORMATION AND EMOTIONAL SIGNALS

A food processor with recognition ability of emotion-related information and emotional signals is disclosed, which comprises: an emotion recognition module and a food processing module. The emotion recognition module is capable of receiving sound signals so as to identify an emotion containing in the received sound signals. The food processing module is capable of producing food products with a taste corresponding to the emotion recognition result of the emotion recognition module.

Discrimination of Three Emotions using Parameters of Autonomic Nervous System Response

Objective: The aim of this study is to compare results of emotion recognition by several algorithms which classify three different emotional states(happiness, neutral, and surprise) using physiological features. Background: Recent emotion recognition studies have tried to detect human emotion by using physiological signals. It is important for emotion recognition to apply on human-computer interaction system for emotion detection. Method: 217 students participated in this experiment. While three kinds of emotional stimuli were presented to participants, ANS responses(EDA, SKT, ECG, RESP, and PPG) as physiological signals were measured in twice first one for 60 seconds as the baseline and 60 to 90 seconds during emotional states. The obtained signals from the session of the baseline and of the emotional states were equally analyzed for 30 seconds. Participants rated their own feelings to emotional stimuli on emotional assessment scale after presentation of emotional stimuli. The emotion classification was analyzed by Linear Discriminant Analysis(LDA, SPSS 15.0), Support Vector Machine (SVM), and Multilayer perceptron(MLP) using difference value which subtracts baseline from emotional state. Results: The emotional stimuli had 96% validity and 5.8 point efficiency on average. There were significant differences of ANS responses among three emotions by statistical analysis. The result of LDA showed that an accuracy of classification in three different emotions was 83.4%. And an accuracy of three emotions classification by SVM was 75.5% and 55.6% by MLP. Conclusion: This study confirmed that the three emotions can be better classified by LDA using various physiological features than SVM and MLP. Further study may need to get this result to get more stability and reliability, as comparing with the accuracy of emotions classification by using other algorithms. Application: This could help get better chances to recognize various human emotions by using physiological signals as well as be applied on human-computer interaction system for recognizing human emotions.

Utterance independent bimodal emotion recognition in spontaneous communication

Emotion expressions sometimes are mixed with the utterance expression in spontaneous face-to-face communication, which makes difficulties for emotion recognition. This article introduces the methods of reducing the utterance influences in visual parameters for the audio-visual-based emotion recognition. The audio and visual channels are first combined under a Multistream Hidden Markov Model (MHMM). Then, the utterance reduction is finished by finding the residual between the real visual parameters and the outputs of the utterance related visual parameters. This article introduces the Fused Hidden Markov Model Inversion method which is trained in the neutral expressed audio-visual corpus to solve the problem. To reduce the computing complexity the inversion model is further simplified to a Gaussian Mixture Model (GMM) mapping. Compared with traditional bimodal emotion recognition methods (e.g., SVM, CART, Boosting), the utterance reduction method can give better results of emotion recognition. The experiments also show the effectiveness of our emotion recognition system when it was used in a live environment.

Novel acoustic features for speech emotion recognition

This paper focuses on acoustic features that effectively improve the recognition of emotion in human speech. The novel features in this paper are based on spectral-based entropy parameters such as fast Fourier transform (FFT) spectral entropy, delta FFT spectral entropy, Mel-frequency filter bank (MFB) spectral entropy, and Delta MFB spectral entropy. Spectral-based entropy features are simple. They reflect frequency characteristic and changing characteristic in frequency of speech. We implement an emotion rejection module using the probability distribution of recognized-scores and rejected-scores. This reduces the false recognition rate to improve overall performance. Recognized-scores and rejected-scores refer to probabilities of recognized and rejected emotion recognition results, respectively. These scores are first obtained from a pattern recognition procedure. The pattern recognition phase uses the Gaussian mixture model (GMM). We classify the four emotional states as anger, sadness, happiness and neutrality. The proposed method is evaluated using 45 sentences in each emotion for 30 subjects, 15 males and 15 females. Experimental results show that the proposed method is superior to the existing emotion recognition methods based on GMM using energy, Zero Crossing Rate (ZCR), linear prediction coefficient (LPC), and pitch parameters. We demonstrate the effectiveness of the proposed approach. One of the proposed features, combined MFB and delta MFB spectral entropy improves performance approximately 10% compared to the existing feature parameters for speech emotion recognition methods. We demonstrate a 4% performance improvement in the applied emotion rejection with low confidence score.