Facial Expression Recognition Task Research Articles

Fusing HOG and convolutional neural network spatial–temporal features for video‐based facial expression recognition

Video-based facial expression recognition (VFER) is the fundamental feature of various computer vision applications. Visual features are the key factors for facial expression recognition. However, the gap between the visual features and the emotions is large. In order to bridge the gap, the proposed method utilises convolutional neural networks (CNNs) and histogram of oriented gradient (HOG) to obtain the more comprehensive feature for VFER. Firstly, it extracts shallow features from the video frame through a number of convolutional kernels in CNNs, which has the characteristics of displacement, scale and deformation invariance. Then, the HOG is employed to extract HOG features from CNN's shallow features, which are strongly correlated with facial expressions. Finally, the support vector machine (SVM) is employed to conduct the task of facial expression recognition. The extensive experiments on RML, CK+ and AFEW5.0 database show that this framework takes on the promising performance and outperforming the state of the arts.

Multi-Modal Recurrent Attention Networks for Facial Expression Recognition

Recent deep neural networks based methods have achieved state-of-the-art performance on various facial expression recognition tasks. Despite such progress, previous researches for facial expression recognition have mainly focused on analyzing color recording videos only. However, the complex emotions expressed by people with different skin colors under different lighting conditions through dynamic facial expressions can be fully understandable by integrating information from multi-modal videos. We present a novel method to estimate dimensional emotion states, where color, depth, and thermal recording videos are used as a multi-modal input. Our networks, called multi-modal recurrent attention networks (MRAN), learn spatiotemporal attention volumes to robustly recognize the facial expression based on attention-boosted feature volumes. We leverage the depth and thermal sequences as guidance priors for color sequence to selectively focus on emotional discriminative regions. We also introduce a novel benchmark for multi-modal facial expression recognition, termed as multi-modal arousal-valence facial expression recognition (MAVFER), which consists of color, depth, and thermal recording videos with corresponding continuous arousal-valence scores. The experimental results show that our method can achieve the state-of-the-art results in dimensional facial expression recognition on color recording datasets including RECOLA, SEWA and AFEW, and a multi-modal recording dataset including MAVFER.

Weakly Supervised Local-Global Attention Network for Facial Expression Recognition

Combining global and local features is an essential solution to improve discriminative performances in facial expression recognition tasks. The limitations of existing methods are that they cannot extract crucial local features and ignore the complementary effects of local and global features. To address these problems, this paper proposes a Weakly Supervised Local-Global Attention Network (WS-LGAN), which uses the attention mechanism to deal with part location and feature fusion problems. Firstly, an Attention Map Generator is designed to get a set of attention maps under weak supervision. It mimics the attention mechanism of human brain and quickly finds the local regions-of-interest. Secondly, bilinear attention pooling is employed to generate and refine local features based on attention maps. Thirdly, a building block called Selective Feature Unit is designed. It allows adaptive weighted fusion of global and local features before making classification. In WS-LGAN, global and local features represent expressions from different aspects. Compared with methods relying on single type of feature, it benefits from local-global complementary advantages. Additionally, contrastive loss is introduced for both local and global features to increase inter-class dispersion and intra-class compactness under different granularities. Experiments on three popular facial expression datasets, including two lab-controlled facial expression datasets and one real-world facial expression dataset show that WS-LGAN achieves state-of-the-art performance, which demonstrates our superiority in facial expression recognition.

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

Facial Expression Recognition (FER) is a challenging task that improves natural human-computer interaction. This paper focuses on automatic FER on a single in-the-wild (ITW) image. ITW images suffer real problems of pose, direction, and input resolution. In this study, we propose a pyramid with super-resolution (PSR) network architecture to solve the ITW FER task. We also introduce a prior distribution label smoothing (PDLS) loss function that applies the additional prior knowledge of the confusion about each expression in the FER task. Experiments on the three most popular ITW FER datasets showed that our approach outperforms all the state-of-the-art methods.

Performance analysis of distance teaching classroom based on machine learning and virtual reality

In remote intelligent teaching, the facial expression features can be recorded in time through facial recognition, which is convenient for teachers to judge the learning status of students in time and helps teachers to change teaching strategies in a timely manner. Based on this, this study applies machine learning and virtual reality technology to distance classroom teaching. Moreover, this study uses different channels to automatically learn global and local features related to facial expression recognition tasks. In addition, this study integrates the soft attention mechanism into the proposed model so that the model automatically learns the feature maps that are more important for facial expression recognition and the salient regions within the feature maps. At the same time, this study performs weighted fusion on the features extracted from different branches, and uses the fused features to re-recognize student features. Finally, this study analyzes the results of this paper through control experiments. The research results show that the algorithm proposed in this paper has good performance and can be applied to the distance teaching system.

Fast and Efficient Facial Expression Recognition Using a Gabor Convolutional Network

Automatic facial expression recognition (FER) is a fundamental topic in computer vision. Many studies have indicated that facial emotion changes are strongly related to certain regions of interest (ROIs), such as the mouth, eyes, eyebrows, and nose; therefore, the features of these facial ROIs are very important for identifying expressions. Since Gabor filters are very efficient in extracting visual content, Gabor orientation filters (GoFs) modulated by Gabor kernels and traditional convolutional filters can capture such ROI information better than conventional convolutional filters. Consequently, this letter presents a light Gabor convolutional network (GCN) consisting of only four Gabor convolutional layers and two linear layers for FER tasks. Extensive experiments on the FER2013, FERPlus and Real-world Affective Faces (RAF) databases demonstrate that the proposed method achieves good recognition accuracy and requires very low computational costs. The source code can be found at https://github.com/general515/Facial_Expression_Recognition_Using _GCN.

Geometry Guided Pose-invariant Facial Expression Recognition.

Driven by recent advances in human-centered computing, Facial Expression Recognition (FER) has attracted significant attention in many applications. However, most conventional approaches either perform face frontalization on a non-frontal facial image or learn separate classifier for each pose. Different from existing methods, this paper proposes an end-to-end deep learning model that allows to simultaneous facial image synthesis and pose-invariant facial expression recognition by exploiting shape geometry of the face image. The proposed model is based on generative adversarial network (GAN) and enjoys several merits. First, given an input face and a target pose and expression designated by a set of facial landmarks, an identity-preserving face can be generated through guiding by the target pose and expression. Second, the identity representation is explicitly disentangled from both expression and pose variations through the shape geometry delivered by facial landmarks. Third, our model can automatically generate face images with different expressions and poses in a continuous way to enlarge and enrich the training set for the FER task. Our approach is demonstrated to perform well when compared with state-of-the-art algorithms on both controlled and in-the-wild benchmark datasets including Multi-PIE, BU-3DFE, and SFEW.

Weakly supervised facial expression recognition via transferred DAL-CNN and active incremental learning

In recent years, facial expression recognition (FER) has becoming a growing topic in computer vision with promising applications on virtual reality and human–robot interaction. Due to the influence of illumination, individual differences, attitude variation, etc., facial expression recognition with robust accuracy in complex environment is still an unsolved problem. Meanwhile, with the wide use of social communication, massive data are uploaded to the Internet; the effective utilization of those data is still a challenge due to noisy label phenomenon in the study of FER. To resolve the above-mentioned problems, firstly, a double active layer-based CNN is established to recognize the facial expression with high accuracy by learning robust and discriminative features from the data, which could enhance the robustness of network. Secondly, an active incremental learning method was utilized to tackle the problem of using Internet data. During the training phase, a two-stage transfer learning method is explored to transfer the relative information from face recognition to FER task to alleviate the inadequate training data in deep convolution network. Besides, in order to make better use of facial expression data from Web site and further improve the FER accuracy, Unconstrained Facial Expression Database from Web site database is built in this paper. Extensive experiments performed on two public facial expression recognition databases FER 2013 and SFEW 2.0 have demonstrated that the proposed scheme outperforms the state-of-the-art methods, which could achieve 67.08% and 51.90%, respectively.

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.

Weighted Patch-based Manifold Regularization Dictionary Pair Learning model for facial expression recognition using Iterative Optimization Classification Strategy

This paper presents a novel method for solving Facial Expression Recognition (FER) task which integrates Manifold Regularization Dictionary Pair Learning and Intra-class Variation Reduced Feature (MRDPL-IVRF) model and Iterative Optimization Classification Strategy (IOCS). In the proposed framework, we firstly divide the facial image into overlapping patches. Weighted Patch-based Local Binary Patterns (WPLBP) is proposed for feature extraction by cascading the weighted LBP features from each patch, which highlights the informative facial regions. MRDPL-IVRF model is then adopted for robust sparse representation of facial features, where IVRFs are generated from the difference between query images and their corresponding approximation of each expression class, which can reduce the intra-class variations resulted by illumination and identity. Finally, IOCS is presented for expression classification which can explore the most unlikely class according to the maximum reconstruction residual. Different from traditional sparse coding classification scheme, our IOCS method iteratively removes irrelevant samples to obtain a more precise representation model, thus can further improve the recognition performance. The feasibility of our framework has been successfully tested on CK, CK+, SEFW, CMU-PIE and Multi-PIE databases. Furthermore, the experiments combining deep features shows that our model can be integrated with existing deep models and obtain better FER results.

Neural response to emotional faces in monozygotic twins: association with familial risk of affective disorders

Aberrant neural and cognitive response to emotional faces has been observed in people at familial risk of an affective disorder. In this functional MRI study of monozygotic twins, we explored neural correlates of the attentional avoidance of emotional faces that we had previously observed in high-risk versus affected twins, and whether an abnormal neural response to emotional faces represents a risk endophenotype. We recruited unaffected monozygotic twins with a co-twin history of mood episodes (high-risk), monozygotic twins with previous mood episodes (affected) and monozygotic twins with no personal or first-degree history of mood episodes (low-risk) between December 2014 and January 2017 based on a nationwide register linkage. Participants viewed fearful and happy faces while performing a gender discrimination task during functional MRI (fMRI) and performed emotional faces dot-probe and facial expression recognition tasks outside the scanner. A total of 129 monozygotic twins underwent whole-brain fMRI. Highrisk twins (n = 38) displayed greater medial and superior prefrontal response to emotional faces than affected twins (n = 62). This greater activity correlated with stronger attentional avoidance of emotional faces in high-risk twins. In contrast, high-risk and affected twins showed no aberrant neural activity to emotional faces compared with low-risk twins (n = 29). A limitation of this study was its cross-sectional design. Greater recruitment of the medial and superior prefrontal cortex during implicit emotion processing in high-risk versus affected twins may represent a compensatory or resilience mechanism. In contrast, aberrant neural response to emotional faces does not seem to be a risk endophenotype for affective disorders.

Comparing ensemble strategies for deep learning: An application to facial expression recognition

Recent works have shown that Convolutional Neural Networks (CNNs), because of their effectiveness in feature extraction and classification tasks, are suitable tools to address the Facial Expression Recognition (FER) problem. Further, it has been pointed out how ensembles of CNNs allow improving classification accuracy. Nevertheless, a detailed experimental analysis on how ensembles of CNNs could be effectively generated in the FER context has not been performed yet, although it would have considerable value for improving the results obtained in the FER task. This paper aims to present an extensive investigation on different aspects of the ensemble generation, focusing on the factors that influence the classification accuracy on the FER context. In particular, we evaluate several strategies for the ensemble generation, different aggregation schemes, and the dependence upon the number of base classifiers in the ensemble. The final objective is to provide some indications for building up effective ensembles of CNNs. Specifically, we observed that exploiting different sources of variability is crucial for the improvement of the overall accuracy. To this aim, pre-processing and pre-training procedures are able to provide a satisfactory variability across the base classifiers, while the use of different seeds does not appear as an effective solution. Bagging ensures a high ensemble gain, but the overall accuracy is limited by poor-performing base classifiers. The impact of increasing the ensemble size specifically depends on the adopted strategy, but also in the best case the performance gain obtained by involving additional base classifiers becomes not significant beyond a certain limit size, thus suggesting to avoid very large ensembles. Finally, the classic averaging voting proves to be an appropriate aggregation scheme, achieving accuracy values comparable to or slightly better than the other experimented operators.

Recognition of dynamic and static facial expressions of emotion among older adults with major depression.

The recognition of facial expressions of emotion is essential to living in society. However, individuals with major depression tend to interpret information considered imprecise in a negative light, which can exert a direct effect on their capacity to decode social stimuli. To compare basic facial expression recognition skills during tasks with static and dynamic stimuli in older adults with and without major depression. Older adults were selected through a screening process for psychiatric disorders at a primary care service. Psychiatric evaluations were performed using criteria from the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5). Twenty-three adults with a diagnosis of depression and 23 older adults without a psychiatric diagnosis were asked to perform two facial emotion recognition tasks using static and dynamic stimuli. Individuals with major depression demonstrated greater accuracy in recognizing sadness (p=0.023) and anger (p=0.024) during the task with static stimuli and less accuracy in recognizing happiness during the task with dynamic stimuli (p=0.020). The impairment was mainly related to the recognition of emotions of lower intensity. The performance of older adults with depression in facial expression recognition tasks with static and dynamic stimuli differs from that of older adults without depression, with greater accuracy regarding negative emotions (sadness and anger) and lower accuracy regarding the recognition of happiness.

What do error patterns in processing facial expressions, social interaction scenes and vocal prosody tell us about the way social cognition works in children with 22q11.2DS?

Impairments in social cognition have been frequently described in 22q11.2 deletion syndrome (22q11.2DS) and are thought to be a hallmark of difficulties in social interactions. The present study addresses aspects that are critical for everyday social cognitive functioning but have received little attention so far. Sixteen children with 22q11.2DS and 22 controls completed 1 task of facial expression recognition, 1 task of attribution of facial expressions to faceless characters involved in visually presented social interactions, and 1 task of attribution of facial expressions to characters involved in aurally presented dialogues. All three tasks have in common to involve processing of emotions. All participants also completed two tasks of attention and two tasks of visual spatial perception, and their parents completed some scales regarding behavioural problems of their children. Patients performed worse than controls in all three tasks of emotion processing, and even worse in the second and third tasks. However, they performed above chance level in all three tasks, and the results were independent of IQ, age and gender. The analysis of error patterns suggests that patients tend to coarsely categorize situations as either attractive or repulsive and also that they have difficulties in differentiating emotions that are associated with threats. An isolated association between the tasks of emotion and behaviour was found, showing that the more frequently patients with 22q11.2DS perceive happiness where there is not, the less they exhibit aggressive behaviour.

Advancing Ensemble Learning Performance through data transformation and classifiers fusion in granular computing context

Classification is a special type of machine learning tasks, which is essentially achieved by training a classifier that can be used to classify new instances. In order to train a high performance classifier, it is crucial to extract representative features from raw data, such as text and images. In reality, instances could be highly diverse even if they belong to the same class, which indicates different instances of the same class could represent very different characteristics. For example, in a facial expression recognition task, some instances may be better described by Histogram of Oriented Gradients features, while others may be better presented by Local Binary Patterns features. From this point of view, it is necessary to adopt ensemble learning to train different classifiers on different feature sets and to fuse these classifiers towards more accurate classification of each instance. On the other hand, different algorithms are likely to show different suitability for training classifiers on different feature sets. It shows again the necessity to adopt ensemble learning towards advances in the classification performance. Furthermore, a multi-class classification task would become increasingly more complex when the number of classes is increased, i.e. it would lead to the increased difficulty in terms of discriminating different classes. In this paper, we propose an ensemble learning framework that involves transforming a multi-class classification task into a number of binary classification tasks and fusion of classifiers trained on different feature sets by using different learning algorithms. We report experimental studies on a UCI data set on Sonar and the CK+ data set on facial expression recognition. The results show that our proposed ensemble learning approach leads to considerable advances in classification performance, in comparison with popular learning approaches including decision tree ensembles and deep neural networks. In practice, the proposed approach can be used effectively to build an ensemble of ensembles acting as a group of expert systems, which show the capability to achieve more stable performance of pattern recognition, in comparison with building a single classifier that acts as a single expert system.

Emotion recognition from facial expression using deep convolutional neural network

Automatic facial expression recognition is an actively emerging research in Emotion Recognition. This paper extends the deep Convolutional Neural Network (CNN) approach to facial expression recognition task. This task is done by detecting the occurrence of facial Action Units (AUs) as a subpart of Facial Action Coding System (FACS) which represents human emotion. In the CNN fully-connected layers we employ a regularization method called “dropout” that proved to be very effective to reduce overfitting. This research uses the extended Cohn Kanade (CK+) dataset which is collected for facial expression recognition experiment. The system performance gain average accuracy rate of 92.81%. The system has been successfully classified eight basic emotion classes. Thus, the proposed method is proven to be effective for emotion recognition.

Emotional Biases and Recurrence in Major Depressive Disorder. Results of 2.5 Years Follow-Up of Drug-Free Cohort Vulnerable for Recurrence.

An interesting factor explaining recurrence risk in Major Depressive Disorder (MDD) may be neuropsychological functioning, i.e., processing of emotional stimuli/information. Negatively biased processing of emotional stimuli/information has been found in both acute and (inconclusively) remitted states of MDD, and may be causally related to recurrence of depression. We aimed to investigate self-referent, memory and interpretation biases in recurrently depressed patients in remission and relate these biases to recurrence. We included 69 remitted recurrent MDD-patients (rrMDD-patients), 35–65 years, with ≥2 episodes, voluntarily free of antidepressant maintenance therapy for at least 4 weeks. We tested self-referent biases with an emotional categorization task, bias in emotional memory by free recall of the emotion categorization task 15 min after completing it, and interpretation bias with a facial expression recognition task. We compared these participants with 43 never-depressed controls matched for age, sex and intelligence. We followed the rrMDD-patients for 2.5 years and assessed recurrent depressive episodes by structured interview. The rrMDD-patients showed biases toward emotionally negative stimuli, faster responses to negative self-relevant characteristics in the emotional categorization, better recognition of sad faces, worse recognition of neutral faces with more misclassifications as angry or disgusting faces and less misclassifications as neutral faces (0.001 < p < 0.05). Of these, the number of misclassifications as angry and the overall performance in the emotional memory task were significantly associated with the time to recurrence (p ≤ 0.04), independent of residual symptoms and number of previous episodes. In a support vector machine data-driven model, prediction of recurrence-status could best be achieved (relative to observed recurrence-rate) with demographic and childhood adversity parameters (accuracy 78.1%; 1-sided p = 0.002); neuropsychological tests could not improve this prediction. Our data suggests a persisting (mood-incongruent) emotional bias when patients with recurrent depression are in remission. Moreover, these persisting biases might be mechanistically important for recurrence and prevention thereof.

Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition

Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. In this paper, we present a novel model, named Deep Attentive Multi-path Convolutional Neural Network (DAM-CNN), for FER. Different from most existing models, DAM-CNN can automatically locate expression-related regions in an expressional image and yield a robust image representation for FER. The proposed model contains two novel modules: an attention-based Salient Expressional Region Descriptor (SERD) and the Multi-Path Variation-Suppressing Network (MPVS-Net). SERD can adaptively estimate the importance of different image regions for FER task, while MPVS-Net disentangles expressional information from irrelevant variations. By jointly combining SERD and MPVS-Net, DAM-CNN is able to highlight expression-relevant features and generate a variation-robust representation for expression classification. Extensive experimental results on both constrained datasets (CK+, JAFFE, TFEID) and unconstrained datasets (SFEW, FER2013, BAUM-2i) demonstrate the effectiveness of our DAM-CNN model.

An efficient density-based clustering with side information and active learning: A case study for facial expression recognition task

An efficient density-based clustering with side information and active learning: A case study for facial expression recognition task

Abstract # 3184 Low grade inflammation induces negative bias in categorization of emotional information

Negative processing biases towards valenced information are a hallmark of depression. We set out to test whether peripheral inflammation can induce such negative interpretation biases. Forty healthy adult men received Salmonella Typhi vaccination and placebo injections on two separate days, using a double-blind crossover design. Blood samples were taken at baseline, 5.5h and 8h post-injection. The participants completed the P1vital® Oxford Emotional Test Battery, which included a Facial Expression Recognition Task (in which a range of intensities of six emotional facial expressions are classified) and Emotional Categorisation and Recall tasks (in which personality descriptors are categorized as positive or negative, and also later recalled). Vaccination increased Interleukin-6 from 1.18 to a peak of 5.21pg/mL at 5.5h post injection (p