Facial Expression Signals Research Articles

Synch-Graph: Multisensory Emotion Recognition Through Neural Synchrony via Graph Convolutional Networks

Human emotions are essentially multisensory, where emotional states are conveyed through multiple modalities such as facial expression, body language, and non-verbal and verbal signals. Therefore having multimodal or multisensory learning is crucial for recognising emotions and interpreting social signals. Existing multisensory emotion recognition approaches focus on extracting features on each modality, while ignoring the importance of constant interaction and co-learning between modalities. In this paper, we present a novel bio-inspired approach based on neural synchrony in audio-visual multisensory integration in the brain, named Synch-Graph. We model multisensory interaction using spiking neural networks (SNN) and explore the use of Graph Convolutional Networks (GCN) to represent and learn neural synchrony patterns. We hypothesise that modelling interactions between modalities will improve the accuracy of emotion recognition. We have evaluated Synch-Graph on two state-of-the-art datasets and achieved an overall accuracy of 98.3% and 96.82%, which are significantly higher than the existing techniques.

Human emotion recognition by optimally fusing facial expression and speech feature

Emotion recognition is a hot research in modern intelligent systems. The technique is pervasively used in autonomous vehicles, remote medical service, and human–computer interaction (HCI). Traditional speech emotion recognition algorithms cannot be effectively generalized since both training and testing data are from the same domain, which have the same data distribution. In practice, however, speech data is acquired from different devices and recording environments. Thus, the data may differ significantly in terms of language, emotional types and tags. To solve such problem, in this work, we propose a bimodal fusion algorithm to realize speech emotion recognition, where both facial expression and speech information are optimally fused. We first combine the CNN and RNN to achieve facial emotion recognition. Subsequently, we leverage the MFCC to convert speech signal to images. Therefore, we can leverage the LSTM and CNN to recognize speech emotion. Finally, we utilize the weighted decision fusion method to fuse facial expression and speech signal to achieve speech emotion recognition. Comprehensive experimental results have demonstrated that, compared with the uni-modal emotion recognition, bimodal features-based emotion recognition achieves a better performance.

Expression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder

Emotion recognition has shown many valuable roles in people's lives under the background of artificial intelligence technology. However, most existing emotion recognition methods have poor recognition performance, which prevents their promotion in practical applications. To alleviate this problem, we proposed an expression-EEG interaction multi-modal emotion recognition method using a deep automatic encoder. Firstly, decision tree is applied as objective feature selection method. Then, based on the facial expression features recognized by sparse representation, the solution vector coefficients are analyzed to determine the facial expression category of the test samples. After that, the bimodal deep automatic encoder is adopted to fuse the EEG signals and facial expression signals. The third layer of BDAE extracts features for training of supervised learning. Finally, LIBSVM classifier is used to complete classification task. We carried out experiments on a constructed video library to verify the proposed emotion recognition method. The results show that the proposed method can effectively extract and integrate high-level emotion-related features in EEG and facial expression signals. The recognition rate of discrete emotion state type and the average emotion recognition rate have been improved relatively, in which the average emotion recognition rate is 85.71%. Overall, the emotion recognition ability has been greatly improved.

Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus

The recognition of fine-grained emotions (i.e., happiness, sad, etc.) has shown its importance in a real-world implementation. The emotion recognition using physiological signals is a challenging task due to the precision of the labelled data while using facial expressions is less appropriate for the real environment. This work proposes a framework for fusing physiological signals and facial expressions modalities to improve classification performance. The feature-level fusion (FLF) and decision-level fusion (DLF) techniques are explored in this work to recognise seven fine-grained emotions. The performance of the proposed framework is evaluated using 34 subjects' data. Our result shows that the fusion of the multiple modalities could improve the overall accuracy compared to the unimodal system by 11.66% and 13.63% for facial expression and physiological signals, respectively. Our work achieved a 73.23% accuracy for seven emotions which is considerable accuracy for the spontaneous emotion corpus.

Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus

The recognition of fine-grained emotions (i.e., happiness, sad, etc.) has shown its importance in a real-world implementation. The emotion recognition using physiological signals is a challenging task due to the precision of the labelled data while using facial expressions is less appropriate for the real environment. This work proposes a framework for fusing physiological signals and facial expressions modalities to improve classification performance. The feature-level fusion (FLF) and decision-level fusion (DLF) techniques are explored in this work to recognise seven fine-grained emotions. The performance of the proposed framework is evaluated using 34 subjects' data. Our result shows that the fusion of the multiple modalities could improve the overall accuracy compared to the unimodal system by 11.66% and 13.63% for facial expression and physiological signals, respectively. Our work achieved a 73.23% accuracy for seven emotions which is considerable accuracy for the spontaneous emotion corpus.

Quantifying Facial Expression Intensity and Signal Use in Deaf Signers.

We live in a world of rich dynamic multisensory signals. Hearing individuals rapidly and effectively integrate multimodal signals to decode biologically relevant facial expressions of emotion. Yet, it remains unclear how facial expressions are decoded by deaf adults in the absence of an auditory sensory channel. We thus compared early and profoundly deaf signers (n= 46) with hearing nonsigners (n= 48) on a psychophysical task designed to quantify their recognition performance for the six basic facial expressions of emotion. Using neutral-to-expression image morphs and noise-to-full signal images, we quantified the intensity and signal levels required by observers to achieve expression recognition. Using Bayesian modeling, we found that deaf observers require more signal and intensity to recognize disgust, while reaching comparable performance for the remaining expressions. Our results provide a robust benchmark for the intensity and signal use in deafness and novel insights into the differential coding of facial expressions of emotion between hearing and deaf individuals.

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition

Recently, the study of emotion recognition has received increasing attentions by the rapid development of noninvasive sensor technologies, machine learning algorithms and compute capability of computers. Compared with single modal emotion recognition, the multimodal paradigm introduces complementary information for emotion recognition. Hence, in this work, we presented a decision level fusion framework for detecting emotions continuously by fusing the Electroencephalography (EEG) and facial expressions. Three types of movie clips (positive, negative, and neutral) were utilized to elicit specific emotions of subjects, the EEG and facial expression signals were recorded simultaneously. The power spectrum density (PSD) features of EEG were extracted by time-frequency analysis, and then EEG features were selected for regression. For the facial expression, the facial geometric features were calculated by facial landmark localization. Long short-term memory networks (LSTM) were utilized to accomplish the decision level fusion and captured temporal dynamics of emotions. The results have shown that the proposed method achieved outstanding performance for continuous emotion recognition, and it yields 0.625± 0.029 of concordance correlation coefficient (CCC). From the results, the fusion of two modalities outperformed EEG and facial expression separately. Furthermore, different numbers of time-steps of LSTM was applied to analyze the temporal dynamic capturing.

Emotion-Specific Facial Activation Maps Based on Infrared Thermal Image Sequences

Research on emotion recognition has recently started to gain increased attention. Inner emotions or thought activity can be determined by analyzing facial expression, behavioral responses, audio, and physiological signals. Facial expression is now recognized as an important form of non-verbal interaction. In this paper, emotion-specific activation maps were constructed to establish infrared thermal facial image sequences as an alternative approach to the determination of the correlation between emotional triggers and changes in facial temperature. During the testing process, data stored in the International Affective Picture System were used to create emotional clips that triggered three different types of emotion in the subjects, and their infrared thermal facial image sequences were simultaneously recorded. For processing, an image calibration protocol was first employed to reduce the variance produced by irregular micro-shifts in the faces of the subjects, followed by independent component analysis and statistical analysis protocols to create the facial emotional activation maps. The test results showed that the problem of selecting local regions when analyzing frame temperature had been resolved. The emotion-specific facial activation maps provide visualized results that facilitate the observation and understanding of information.

Quantifying facial expression signal and intensity use during development

Behavioral studies investigating facial expression recognition during development have applied various methods to establish by which age emotional expressions can be recognized. Most commonly, these methods employ static images of expressions at their highest intensity (apex) or morphed expressions of different intensities, but they have not previously been compared. Our aim was to (a) quantify the intensity and signal use for recognition of six emotional expressions from early childhood to adulthood and (b) compare both measures and assess their functional relationship to better understand the use of different measures across development. Using a psychophysical approach, we isolated the quantity of signal necessary to recognize an emotional expression at full intensity and the quantity of expression intensity (using neutral expression image morphs of varying intensities) necessary for each observer to recognize the six basic emotions while maintaining performance at 75%. Both measures revealed that fear and happiness were the most difficult and easiest expressions to recognize across age groups, respectively, a pattern already stable during early childhood. The quantity of signal and intensity needed to recognize sad, angry, disgust, and surprise expressions decreased with age. Using a Bayesian update procedure, we then reconstructed the response profiles for both measures. This analysis revealed that intensity and signal processing are similar only during adulthood and, therefore, cannot be straightforwardly compared during development. Altogether, our findings offer novel methodological and theoretical insights and tools for the investigation of the developing affective system.

Objective Pain Measurement based on Physiological Signals

In an attempt to understand human physiological signals when an individual is subjected to pain, we set up a tonic pain experiment in a laboratory setting. The subjects’ physiological signals were recorded, timestamped, and compared to an initial 30 second baseline measurement. Subjects were also asked to verbally state their level of pain based on a visual analog scale in order to compare reported pain levels with physiological signals. The physiological signals measured were: Electroencephalography (EEG), Pupillary Unrest Under Ambient Light (PUAL), Skin Conductance (SC), Electromyography (EMG), Respiration Rate (RR), Blood Volume Pulse (BVP), Skin Temperature (ST), Blood Pressure (BP), and Facial Expression (FE). ANOVA and frequency domain analyses were conducted on the data in order to determine whether there was a significant difference between the ‘pain’ and ‘no pain’ (baseline) states of an individual. Based on our results, skin conductance, PUAL, facial expression, and EEG signals were theorized to be good signals for the classification of tonic pain, or any pain applied directly to an individual.

Emotional state recognition using facial expression, voice and physiological signal

Emotional state recognition using facial expression, voice and physiological signal

A Novel Framework for Estimating Viewer Interest by Unsupervised Multimodal Anomaly Detection

A reliable method to estimate viewer interest is highly sought after for human-centered video information retrieval. A method that estimates viewer interest while users are watching Web videos is presented in this paper. The method uses a framework for anomaly detection based on collaborative use of facial expression and biological signals such as electroencephalogram (EEG) signals. To the best of our knowledge, there have been no studies that have taken into account two actual mechanisms of the behavior of users while they are watching Web videos. First, whereas most Web videos garner very little attention, a small number attract millions of views. Therefore, a framework for anomaly detection is newly applied to facial expression and EEG in order to model the imbalanced distribution of popularity. Second, since the number of Web videos that are labeled by users as interesting/not interesting is generally too small to estimate viewer interest by a supervised approach, the proposed method utilizes parametric techniques for anomaly detection, which estimates viewer interest in an unsupervised way. Unlike some related studies for estimating viewer interest, our method takes into account actual mechanisms of the behavior of users while they are watching Web videos by utilizing parametric techniques for anomaly detection. Then viewer interest can be estimated on the basis of an anomaly score calculated from our proposed method. Consequently, successful estimation of viewer interest based on a framework for anomaly detection, via collaborative use of facial expression and biological signals, becomes feasible.

Emotional State Recognition Using Facial Expression, Voice, and Physiological Signal

Emotion recognition is an important aspect of affective computing, one of whose aims is the study and development of behavioral and emotional interaction between human and machine. In this context, another important point concerns acquisition devices and signal processing tools which lead to an estimation of the emotional state of the user. This article presents a survey about concepts around emotion, multimodality in recognition, physiological activities and emotional induction, methods and tools for acquisition and signal processing with a focus on processing algorithm and their degree of reliability.

A multimodal dataset for various forms of distracted driving

We describe a multimodal dataset acquired in a controlled experiment on a driving simulator. The set includes data for n=68 volunteers that drove the same highway under four different conditions: No distraction, cognitive distraction, emotional distraction, and sensorimotor distraction. The experiment closed with a special driving session, where all subjects experienced a startle stimulus in the form of unintended acceleration—half of them under a mixed distraction, and the other half in the absence of a distraction. During the experimental drives key response variables and several explanatory variables were continuously recorded. The response variables included speed, acceleration, brake force, steering, and lane position signals, while the explanatory variables included perinasal electrodermal activity (EDA), palm EDA, heart rate, breathing rate, and facial expression signals; biographical and psychometric covariates as well as eye tracking data were also obtained. This dataset enables research into driving behaviors under neatly abstracted distracting stressors, which account for many car crashes. The set can also be used in physiological channel benchmarking and multispectral face recognition.

Feature Fusion Algorithm for Multimodal Emotion Recognition from Speech and Facial Expression Signal

In order to overcome the limitation of single mode emotion recognition. This paper describes a novel multimodal emotion recognition algorithm, and takes speech signal and facial expression signal as the research subjects. First, fuse the speech signal feature and facial expression signal feature, get sample sets by putting back sampling, and then get classifiers by BP neural network (BPNN). Second, measure the difference between two classifiers by double error difference selection strategy. Finally, get the final recognition result by the majority voting rule. Experiments show the method improves the accuracy of emotion recognition by giving full play to the advantages of decision level fusion and feature level fusion, and makes the whole fusion process close to human emotion recognition more, with a recognition rate 90.4%.

Characterizing the Manifolds of Dynamic Facial Expression Categorization

Visual categorization seeks to understand how logical combinations of visual cues (e.g. wide left eye and opened mouth) provide singly necessary and jointly sufficient conditions to access categories in memory (e.g. surprise). Such combinations of visual cues constitute the categorization manifold underlying the information goals of face categorization mechanisms and are therefore critical for their understanding. Yet, no method currently exists to reliably characterize the visual cues of the categorization manifold (i.e. its dimensions, such as wide eyes and opened mouth) and how they combine (i.e. the manifold topology which dictates e.g. that wide eyes and opened mouth can be used independently or must be used jointly). Here we present a generic method to characterize categorization manifolds and apply it to observers categorizing dynamic facial expressions of emotion. To generate information, we used the Generative Face Grammar (GFG) platform (Yu et al., 2012) that selects on each trial (N = 2,400 trials/observer) a random set of Action Units (AUs) and values for their parametric activation (Jack et al., 2012). We asked 60 naive Western Caucasian observers to categorize the presented random facial animation according to one of six classic emotions (happy, surprise, fear, disgust, anger, sad, plus don't know). For each observer, we used a Non-negative Matrix Factorization (NMF) algorithm to extract AU x Time components of facial expression signals associated with the pereceptual categorization of each emotion. We then performed a Linear Discriminant Analysis (LDA) to select the components (i.e. manifold dimensions) that discriminate between six emotion categories (Quian Quiroga and Panzeri, 2009; Delis et al., 2013). Hence, the dimensions of the resultant categorization manifolds represent the strategies observers use to categorize the emotions. Our data show that observers use multiple categorization strategies, which constitute the atomic signals of emotion communication via facial expressions.

A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

Affective states, moods and emotions, are an integral part of human nature: they shape our thoughts, govern the behavior of the individual, and influence our interpersonal relationships. The last decades have seen a growing interest in the automatic detection of such states from voice, facial expression, and physiological signals, primarily with the goal of enhancing human-computer interaction with an affective component. With the advent of brain-computer interface research, the idea of affective brain-computer interfaces (aBCI), enabling affect detection from brain signals, arose. In this article, we set out to survey the field of neurophysiology-based affect detection. We outline possible applications of aBCI in a general taxonomy of brain-computer interface approaches and introduce the core concepts of affect and their neurophysiological fundamentals. We show that there is a growing body of literature that evidences the capabilities, but also the limitations and challenges of affect detection from neurophysiological activity.

Dynamic mental models of culture-specific emotions

According to the Universality Hypothesis, facial expressions of emotion comprise a universal set of six basic signals common to all humans (i.e., happy, surprise, fear, disgust, anger, sad; Ekman et al., 1969). In contrast, Jack et al. (2012) demonstrate that although Western Caucasians (WC) represent the six basic emotions with the same dynamic facial movements, East Asians (EA) do not. This raises the questions of (1) what are the basic emotions in the EA culture? and (2) what are the corresponding culture-specific facial signals of emotion transmission? To address the first question, we clustered emotion words in WC (English, 50 observers) and EA (Chinese, 50 observers) culture. Each participant rated on a bipolar scale the pairwise similarity of selected emotion words in their own language (see Methods). We applied clustering analyses to these data. The English clusters comprised 8 emotion categories including the six basic ones, plus pride and shame (e.g., Tracey & Robins, 2004). In contrast, the Chinese clusters showed a more complex structure of 10 basic emotions (see Figure S1). Using the resulting basic emotion categories as response labels, we applied 4-dimensional reverse correlation (Yu et al. 2012) to reconstruct culture-specific face signals that transmit each emotion. As in Jack et al. (2012), on each experimental trial the observer viewed a random facial animation generated by computer graphics platform. Observers interpreted the facial animation as expressive when the facial movements corresponded with their mental representation of that emotion (Figure S2). Reverse correlation analyses produced, for each observer, a dynamic model per basic emotion. Our analyses revealed culture-specific facial expression signals, refuting the University Hypothesis. For the first time, we also derive the cultural face signals that articulate emotion communication in the EA culture (see Figure S3 for examples).

Perceptual weighting of pain behaviours of others, not information integration, varies with expertise

Being able to estimate effort pain in patients is important for health care providers working in physical rehabilitation services. Previous studies have shown that clinicians and physiotherapists underestimate patients' pain. The present study examined how two sources of visual information, namely body kinematics (movement speed and postural constraints) and facial expressions are integrated in order to estimate effort pain magnitude experienced by a paraplegic person performing a sitting pivot transfer. In addition, the effect of familiarity with paraplegia on judgment was assessed by comparing performance among physiotherapists, paraplegic patients and unfamiliar participants. Functional measurement was used to determine the psychophysical law of visual information integration carried by pain behaviours (guarding and facial expression). Results indicate that guarding behaviour (specified by movement speed) carried important information for perceived effort pain independently of familiarity. In contrast, facial expression of pain was relevant only to unfamiliar and physiotherapist participants and not to paraplegic participants. Even if physiotherapists underestimated effort pain as compared to the other groups, they relied more strongly on facial expression, than other participants, in their estimation of effort pain expressed by a paraplegic patient. These results bring further insights into understanding physiotherapists' perception of patients' pain. In order to improve their ability to adapt the difficulty of the rehabilitation sessions, they should learn to raise their global level of pain magnitude estimation (for example, by performing themselves the body movement) instead of overweighting facial pain expression signals.