Automatic Emotion Detection Research Articles

Development of Filtered Bispectrum for EEG Signal Feature Extraction in Automatic Emotion Recognition Using Artificial Neural Networks

The development of automatic emotion detection systems has recently gained significant attention due to the growing possibility of their implementation in several applications, including affective computing and various fields within biomedical engineering. Use of the electroencephalograph (EEG) signal is preferred over facial expression, as people cannot control the EEG signal generated by their brain; the EEG ensures a stronger reliability in the psychological signal. However, because of its uniqueness between individuals and its vulnerability to noise, use of EEG signals can be rather complicated. In this paper, we propose a methodology to conduct EEG-based emotion recognition by using a filtered bispectrum as the feature extraction subsystem and an artificial neural network (ANN) as the classifier. The bispectrum is theoretically superior to the power spectrum because it can identify phase coupling between the nonlinear process components of the EEG signal. In the feature extraction process, to extract the information contained in the bispectrum matrices, a 3D pyramid filter is used for sampling and quantifying the bispectrum value. Experiment results show that the mean percentage of the bispectrum value from 5 × 5 non-overlapped 3D pyramid filters produces the highest recognition rate. We found that reducing the number of EEG channels down to only eight in the frontal area of the brain does not significantly affect the recognition rate, and the number of data samples used in the training process is then increased to improve the recognition rate of the system. We have also utilized a probabilistic neural network (PNN) as another classifier and compared its recognition rate with that of the back-propagation neural network (BPNN), and the results show that the PNN produces a comparable recognition rate and lower computational costs. Our research shows that the extracted bispectrum values of an EEG signal using 3D filtering as a feature extraction method is suitable for use in an EEG-based emotion recognition system.

Automatic Emotional State Detection using Facial Expression Dynamic in Videos

In this paper, an automatic emotion detection system is built for a computer or machine to detect the emotional state from facial expressions in human computer communication. Firstly, dynamic motion features are extracted from facial expression videos and then advanced machine learning methods for classification and regression are used to predict the emotional states. The system is evaluated on two publicly available datasets, i.e. GEMEP_FERA and AVEC2013, and satisfied performances are achieved in comparison with the baseline results provided. With this emotional state detection capability, a machine can read the facial expression of its user automatically. This technique can be integrated into applications such as smart robots, interactive games and smart surveillance systems.

Emotional and User-Specific Acoustic Cues for Improved Analysis of Naturalistic Interactions

T Human-Computer Interaction recently received an increased attention. This is not just a matter of making the operation of technical systems as simple as pissuble, but also to enable a possibly natural interaction. In this context, especially the speech-based operation gained an increased attention. For example, modern smart phones and televisions offer a robust voice control, which is attributed to various technical improvements in recent years. Nevertheless, voice control still seems artificial. Only self-contained dialogues with short statements can be managed. Furthermore, just the content of speech is evaluated. The way in which something is said remains unconsidered, although it is known from human communication, that the transmitted emotion is important in order to communicate successfully. A relatively new branch of research, the “Affective Computing”, has, amongst other objectives, the aim to develop technical systems that recognise and interpret emotions and respond to them appropriately. In this case, speech-based automatic emotion recognition has a major role. For emotion recognition, it is important to know how emotions can be presented and how they are expressed. For this purpose, it is helpful to rely on empirical evidences of the psychology of emotions. Unfortunately, there is no uniform representation of emotions. Also the definition of appropriate emotion-distinctive acoustic features is rather descriptive in psychology. Therefore, the automatic detection of emotions is based on proven methods of automatic speech recognition, which have also been shown as appropriate for emotion recognition. Automatic emotion recognition is, as speech recognition, a branch of pattern recognition. Contrary to emotion psychology, it is data-driven, that means insights are gathered from sampled data. For emotion recognition the phases “annotation”, “modelling” and “recognition” are distinguishable. The annotation categorises speech data according to predefined emotion-terms. Modelling generates recognisers to categorise data automatically. Recognition performs a previously unknown allocation of data to emotional classes. In the beginning, automatic emotion recognition was usually based – due to the lack of suitable data sets – on acted and very expressive emotional expressions. In this case, based on features and detection methods known from speech recognition, very good recognition results of over 80% in distinguishing of up to seven emotions could be achieved. However, for human-machine interaction these recognisers were unsuitable because in this case emotions are not that expressive. Therefore, in collaboration with

Path Models of Vocal Emotion Communication

We propose to use a comprehensive path model of vocal emotion communication, encompassing encoding, transmission, and decoding processes, to empirically model data sets on emotion expression and recognition. The utility of the approach is demonstrated for two data sets from two different cultures and languages, based on corpora of vocal emotion enactment by professional actors and emotion inference by naïve listeners. Lens model equations, hierarchical regression, and multivariate path analysis are used to compare the relative contributions of objectively measured acoustic cues in the enacted expressions and subjective voice cues as perceived by listeners to the variance in emotion inference from vocal expressions for four emotion families (fear, anger, happiness, and sadness). While the results confirm the central role of arousal in vocal emotion communication, the utility of applying an extended path modeling framework is demonstrated by the identification of unique combinations of distal cues and proximal percepts carrying information about specific emotion families, independent of arousal. The statistical models generated show that more sophisticated acoustic parameters need to be developed to explain the distal underpinnings of subjective voice quality percepts that account for much of the variance in emotion inference, in particular voice instability and roughness. The general approach advocated here, as well as the specific results, open up new research strategies for work in psychology (specifically emotion and social perception research) and engineering and computer science (specifically research and development in the domain of affective computing, particularly on automatic emotion detection and synthetic emotion expression in avatars).

3D face model dataset: Automatic detection of facial expressions and emotions for educational environments

AbstractEmotion detection using facial images is a technique that researchers have been using for the last two decades to try to analyze a person's emotional state given his/her image. Detection of various kinds of emotion using facial expressions of students in educational environment is useful in providing insight into the effectiveness of tutoring sessions. Robust recognition with conventional 2D cameras is still not possible under realistic conditions, ie, in the presence of variation in lighting and illumination and pose variations of the participants. This paper discusses in brief how facial expression recognition from 3D models could be achieved for applications under various educational environments. 3D Face Model dataset for various facial expressions of about 95 undergraduate students of the Department of Information Science and Engineering was collected. The new dataset produced comprise 3D models of emotions represented by five groups of facial expressions, namely: normal, happy, sad, surprised and angry, which are mainly seen among students during their college days. These data were collected using a Kinect camera, which also gives the depth map for the facial expressions. The dataset collected is a point cloud. The Kinect camera measures a large number of points on the face of the person and this output is stored as a point cloud data (PCD) file. The point cloud thus represents the sets of points that the camera has measured. The data are stored in the PCD file format, which supports the 3D point cloud data, and which can be viewed using any point cloud library (PCL) viewer.

자동 감성 인식을 위한 비교사-교사 분류기의 복합 설계

The emotion is deeply affected by human behavior and cognitive process, so it is important to do research about the emotion. However, the emotion is ambiguous to clarify because of different ways of life pattern depending on each individual characteristics. To solve this problem, we use not only physiological signal for objective analysis but also hybrid unsupervised-supervised learning classifier for automatic emotion detection. The hybrid emotion classifier is composed of K-means, genetic algorithm and support vector machine. We acquire four different kinds of physiological signal including electroencephalography(EEG), electrocardiography(ECG), galvanic skin response(GSR) and skin temperature(SKT) as well as we use 15 features extracted to be used for hybrid emotion classifier. As a result, hybrid emotion classifier(80.6%) shows better performance than SVM(31.3%).

Spatial attention effects of disgusted and fearful faces.

Effective processing of threat-related stimuli is of significant evolutionary advantage. Given the intricate relationship between attention and the neural processing of threat-related emotions, this study manipulated attention allocation and emotional categories of threat-related stimuli as independent factors and investigated the time course of spatial-attention-modulated processing of disgusting and fearful stimuli. The participants were instructed to direct their attention either to the two vertical or to the two horizontal locations, where two faces and two houses would be presented. The task was to respond regarding the physical identity of the two stimuli at cued locations. Event-related potentials (ERP) evidences were found to support a two-stage model of attention-modulated processing of threat-related emotions. In the early processing stage, disgusted faces evoked larger P1 component at right occipital region despite the attention allocation while larger N170 component was elicited by fearful faces at right occipito-temporal region only when participants attended to houses. In the late processing stage, the amplitudes of the parietal P3 component enhanced for both disgusted and fearful facial expressions only when the attention was focused on faces. According to the results, we propose that the temporal dynamics of the emotion-by-attention interaction consist of two stages. The early stage is characterized by quick and specialized neural encoding of disgusting and fearful stimuli irrespective of voluntary attention allocation, indicating an automatic detection and perception of threat-related emotions. The late stage is represented by attention-gated separation between threat-related stimuli and neutral stimuli; the similar ERP pattern evoked by disgusted and fearful faces suggests a more generalized processing of threat-related emotions via top-down attentional modulation, based on which the defensive behavior in response to threat events is largely facilitated.

Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web

Automatic detection of emotions like sarcasm or nastiness in online written conversation is a difficult task. It requires a system that can manage some kind of knowledge to interpret that emotional language is being used. In this work, we try to provide this knowledge to the system by considering alternative sets of features obtained according to different criteria. We test a range of different feature sets using two different classifiers. Our results show that the sarcasm detection task benefits from the inclusion of linguistic and semantic information sources, while nasty language is more easily detected using only a set of surface patterns or indicators.

Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text

A major limitation in the automatic detection of affect, feelings, emotions, sentiments, and opinions in text is the lack of proper differentiation between these subjective terms and understanding of how they relate to one another. This lack of differentiation not only leads to inconsistency in terminology usage but also makes the subtleties and nuances expressed by the five terms difficult to understand, resulting in subpar detection of the terms in text. In light of such limitation, this paper clarifies the differences between these five subjective terms and reveals significant concepts to the computational linguistics community for their effective detection and processing in text.

Automatic Emotional State Detection using Facial Expression Dynamic in Videos

In this paper, an automatic emotion detection system is built for a computer or machine to detect the emotional state from facial expressions in human computer communication. Firstly, dynamic motion features are extracted from facial expression videos and then advanced machine learning methods for classification and regression are used to predict the emotional states. The system is evaluated on two publicly available datasets, i.e. GEMEP_FERA and AVEC2013, and satisfied performances are achieved in comparison with the baseline results provided. With this emotional state detection capability, a machine can read the facial expression of its user automatically. This technique can be integrated into applications such as smart robots, interactive games and smart surveillance systems.

Iterative Feature Normalization Scheme for Automatic Emotion Detection from Speech

The externalization of emotion is intrinsically speaker-dependent. A robust emotion recognition system should be able to compensate for these differences across speakers. A natural approach is to normalize the features before training the classifiers. However, the normalization scheme should not affect the acoustic differences between emotional classes. This study presents the iterative feature normalization (IFN) framework, which is an unsupervised front-end, especially designed for emotion detection. The IFN approach aims to reduce the acoustic differences, between the neutral speech across speakers, while preserving the inter-emotional variability in expressive speech. This goal is achieved by iteratively detecting neutral speech for each speaker, and using this subset to estimate the feature normalization parameters. Then, an affine transformation is applied to both neutral and emotional speech. This process is repeated till the results from the emotion detection system are consistent between consecutive iterations. The IFN approach is exhaustively evaluated using the IEMOCAP database and a data set obtained under free uncontrolled recording conditions with different evaluation configurations. The results show that the systems trained with the IFN approach achieve better performance than systems trained either without normalization or with global normalization.

Emotion detection in suicide notes

The success of suicide prevention, a major public health concern worldwide, hinges on adequate suicide risk assessment. Online platforms are increasingly used for expressing suicidal thoughts, but manual monitoring is unfeasible given the information overload experts are confronted with. We investigate whether the recent advances in natural language processing, and more specifically in sentiment mining, can be used to accurately pinpoint 15 different emotions, which might be indicative of suicidal behavior.A system for automatic emotion detection was built using binary support vector machine classifiers. We hypothesized that lexical and semantic features could be an adequate way to represent the data, as emotions seemed to be lexicalized consistently. The optimal feature combination for each of the different emotions was determined using bootstrap resampling. Spelling correction was applied to the input data, in order to reduce lexical variation.Classification performance varied between emotions, with scores up to 68.86% F-score. F-scores above 40% were achieved for six of the seven most frequent emotions: thankfulness, guilt, love, information, hopelessness and instructions. The most salient features are trigram and lemma bags-of-words and subjectivity clues. Spelling correction had a slightly positive effect on classification performance.We showed that fine-grained automatic emotion detection benefits from classifier optimization and a combined lexico-semantic feature representation. The modest performance improvements obtained through spelling correction might indicate the robustness of the system to noisy input text. We conclude that natural language processing techniques have future application potential for suicide prevention.

Visual mismatch negativity in the detection of facial emotions in patients with panic disorder

Panic disorder is a highly prevalent mental disease characterized by sudden and recurrent panic attacks. Studies on patients with panic disorder have found dysfunctional strategic processing of facial expressions and abnormal automatic processing of panic-related stimuli. However, whether the automatic processing of facial emotions in patients with panic disorder is normal is unknown. Visual mismatch negativity (vMMN) reflects automatic processing of change detection. We recorded expression-related vMMN to compare the automatic detection of schematic facial emotions (negative and positive) of 12 patients with panic disorder and 17 age-matched and education-matched controls. During the late time window (220-330 ms), the vMMN amplitudes for two emotions (negative and positive) were significantly decreased in patients with panic disorder (P=0.02). This finding implies that the ability to automatically detect facial emotions is impaired in patients with panic disorder. This is the first report on expression-related vMMN being used with patients with panic disorder.

Recognition of Human Emotion from a Speech Signal Based on Plutchik's Model

Recognition of Human Emotion from a Speech Signal Based on Plutchik's ModelMachine recognition of human emotional states is an essential part in improving man-machine interaction. During expressive speech the voice conveys semantic message as well as the information about emotional state of the speaker. The pitch contour is one of the most significant properties of speech, which is affected by the emotional state. Therefore pitch features have been commonly used in systems for automatic emotion detection. In this work different intensities of emotions and their influence on pitch features have been studied. This understanding is important to develop such a system. Intensities of emotions are presented on Plutchik's cone-shaped 3D model. ThekNearest Neighbor algorithm has been used for classification. The classification has been divided into two parts. First, the primary emotion has been detected, then its intensity has been specified. The results show that the recognition accuracy of the system is over 50% for primary emotions, and over 70% for its intensities.

Automatic detection of emotions with music files

The amount of music files available on the internet is constantly growing, as well as the access to recordings. Music is now so readily accessible in digital form that personal collections can easily exceed the practical limits of the time we have to listen to them. Today, the problem of building music recommendation systems, including systems which can automatically detect emotions with music files, is of great importance. In this work, we present a new strategy for automatic detection of emotions with musical instrument recordings. We use Thayer’s model to represent emotions. We extract timbre-related acoustic features. We train and test two classifiers. Results yield good recognition accuracy.

Lexicon-based detection of emotion in different types of texts: Preliminary remarks

Paragraphs of four genres are analysed to detect their emotional colouring, while a lexicon-based approach of linguistic analysis is weighed against reader opinion. The aim is to find out the prospects of automatic detection of emotion in any text by using a very small lexicon of about 600 frequent emotion words. DOI: http://dx.doi.org/10.5128/ERYa8.11

Multi-label classification of music by emotion

This work studies the task of automatic emotion detection in music. Music may evoke more than one different emotion at the same time. Single-label classification and regression cannot model this multiplicity. Therefore, this work focuses on multi-label classification approaches, where a piece of music may simultaneously belong to more than one class. Seven algorithms are experimentally compared for this task. Furthermore, the predictive power of several audio features is evaluated using a new multi-label feature selection method. Experiments are conducted on a set of 593 songs with six clusters of emotions based on the Tellegen-Watson-Clark model of affect. Results show that multi-label modeling is successful and provide interesting insights into the predictive quality of the algorithms and features.

A Data-Driven Affective Analysis Framework Toward Naturally Expressive Speech Synthesis

An essential step in the generation of expressive speech synthesis is the automatic detection and classification of emotions most likely to be present in textual input. Though increasingly data-driven, emotion analysis still relies on critical expert knowledge in order to isolate the emotional keywords or keysets necessary to the construction of affective categories. This makes it vulnerable to any discrepancy between the ensuing taxonomy of affective states and the underlying domain of discourse. This paper proposes a more general framework, latent affective mapping, which exploits two separate levels of semantic information: the first one encapsulates the foundations of the domain considered, while the second one specifically accounts for the overall affective fabric of the language. Exposing the emergent relationship between these two levels advantageously steers the emotion classification process. Empirical evidence suggests that this approach is effective for automatic emotion analysis in text. This bodes well for its deployability toward naturally expressive speech synthesis.

Emotion Recognition in Text for 3-D Facial Expression Rendering

Emotions are a key semantic component of human communication. This study focuses on automatic emotion detection in descriptive sentences and how this can be used to tune facial expression parameters for 3-D character generation. A comparison of manual and automatic word feature selection approaches is performed to determine the influence of word features on classification accuracy using support vector machines (SVM). The automatic emotion feature selection algorithm presented here builds on the framework used by mutual information for feature selection. Results of the study indicate that the set of automatically selected features was as good as the set of manually selected features. The proposed automatic feature selection algorithm implemented in this study helped to detect new words from the training corpus which were relevant to the classification task but were not considered by the researchers. An example of potential outcomes from facial expression tuning is also presented. The analysis includes initial results for dealing with the class imbalance challenge present in the data.

Authentic facial expression analysis

There is a growing trend toward emotional intelligence in human–computer interaction paradigms. In order to react appropriately to a human, the computer would need to have some perception of the emotional state of the human. We assert that the most informative channel for machine perception of emotions is through facial expressions in video. One current difficulty in evaluating automatic emotion detection is that there are currently no international databases which are based on authentic emotions. The current facial expression databases contain facial expressions which are not naturally linked to the emotional state of the test subject. Our contributions in this work are twofold: first, we create the first authentic facial expression database where the test subjects are showing the natural facial expressions based upon their emotional state. Second, we evaluate the several promising machine learning algorithms for emotion detection which include techniques such as Bayesian networks, SVMs, and decision trees.