Expression Recognition Accuracy Research Articles

Maternal nurturing experience affects the perception and recognition of adult and infant facial expressions.

The perception and recognition of facial expressions are crucial for parenting. This study investigated whether and how maternal nurturing experience and trait anxiety influence the perception and recognition of infant and adult facial expressions. This was assessed by comparing the performance of primiparous mothers (n = 25) and non-mothers (n = 28) on an emotional face perception task. Trait anxiety was measured using the Japanese version of the State-Trait Anxiety Inventory (STAI). We found that mothers had higher recognition accuracy for facial expressions, but only of adults, not infants. Moreover, as trait anxiety increased, so did mothers’ sensitivity in perceiving facial expressions of both infants and adults. These findings suggest that maternal nurturing experience does enhance the recognition of adult emotional expressions, and an optimal level of maternal trait anxiety may enhance mothers’ sensitivity toward infants’ and adults’ emotional signals.

Exploring emotional expression recognition in aging adults using the Moving Window Technique

Adult aging is associated with difficulties in recognizing negative facial expressions such as fear and anger. However, happiness and disgust recognition is generally found to be less affected. Eye-tracking studies indicate that the diagnostic features of fearful and angry faces are situated in the upper regions of the face (the eyes), and for happy and disgusted faces in the lower regions (nose and mouth). These studies also indicate age-differences in visual scanning behavior, suggesting a role for attention in emotion recognition deficits in older adults. However, because facial features can be processed extrafoveally, and expression recognition occurs rapidly, eye-tracking has been questioned as a measure of attention during emotion recognition. In this study, the Moving Window Technique (MWT) was used as an alternative to the conventional eye-tracking technology. By restricting the visual field to a moveable window, this technique provides a more direct measure of attention. We found a strong bias to explore the mouth across both age groups. Relative to young adults, older adults focused less on the left eye, and marginally more on the mouth and nose. Despite these different exploration patterns, older adults were most impaired in recognition accuracy for disgusted expressions. Correlation analysis revealed that among older adults, more mouth exploration was associated with faster recognition of both disgusted and happy expressions. As a whole, these findings suggest that in aging there are both attentional differences and perceptual deficits contributing to less accurate emotion recognition.

Low-rank sparse coding and region of interest pooling for dynamic 3D facial expression recognition

In this paper, we propose a dynamic three-dimensional facial expression recognition using low-rank sparse codes pooled from automatically detected regions of interests. Low-rank sparse coding has been applied for the first time in dynamic facial expression recognition and a novel region-based pooling is suggested. Twelve regions of interests are defined using detected landmarks based on facial expression activation dynamics. Landmarks are tracked in subsequent frames using multi-point tracker. Temporal segmentation is utilized by using geometric information extracted from detected landmarks. LBP-TOP feature descriptors are extracted from cuboids inside spatiotemporal regions of interests in both texture and depth sequences. Texture and depth features are then fused to form the feature matrix. Low-rank sparse coding is utilized to obtain sparse codes from feature descriptors. Finally, hidden-state conditional random fields are employed to classify six basic expressions. Experimental results and comparison with recent studies verify that proposed method improves the accuracy of dynamic facial expression recognition. The average accuracy of recognition of six basic expressions is improved by the proposed system from 83.07 to 85.12\(\%\) on 95 subjects in BU-4DFE data set.

Incongruence Between Observers' and Observed Facial Muscle Activation Reduces Recognition of Emotional Facial Expressions From Video Stimuli.

According to embodied cognition accounts, viewing others’ facial emotion can elicit the respective emotion representation in observers which entails simulations of sensory, motor, and contextual experiences. In line with that, published research found viewing others’ facial emotion to elicit automatic matched facial muscle activation, which was further found to facilitate emotion recognition. Perhaps making congruent facial muscle activity explicit produces an even greater recognition advantage. If there is conflicting sensory information, i.e., incongruent facial muscle activity, this might impede recognition. The effects of actively manipulating facial muscle activity on facial emotion recognition from videos were investigated across three experimental conditions: (a) explicit imitation of viewed facial emotional expressions (stimulus-congruent condition), (b) pen-holding with the lips (stimulus-incongruent condition), and (c) passive viewing (control condition). It was hypothesised that (1) experimental condition (a) and (b) result in greater facial muscle activity than (c), (2) experimental condition (a) increases emotion recognition accuracy from others’ faces compared to (c), (3) experimental condition (b) lowers recognition accuracy for expressions with a salient facial feature in the lower, but not the upper face area, compared to (c). Participants (42 males, 42 females) underwent a facial emotion recognition experiment (ADFES-BIV) while electromyography (EMG) was recorded from five facial muscle sites. The experimental conditions’ order was counter-balanced. Pen-holding caused stimulus-incongruent facial muscle activity for expressions with facial feature saliency in the lower face region, which reduced recognition of lower face region emotions. Explicit imitation caused stimulus-congruent facial muscle activity without modulating recognition. Methodological implications are discussed.

Emotional expression and micro-expression recognition in depressive patients

Facial expression is an important nonverbal act of human to express their emotion. It is the most prevalent indicator of emotion in human’s emotional study. Recently, how depressive people process the emotional information have received much interest. Previous studies have found that the ability to recognize facial expression in those people with major depressive disorder (MDD) is damaged. For example, they recognize negative expression slower, and show a negative bias. Furthermore, patients with depression have structural or functional abnormalities related to emotion-related brain regions. As a result, depressive patients may not accurately identify subtle changes on facial expression in social interactions. As a special form of subtle change, micro-expression is difficult to detect for healthy people. This is due to the short duration and weakened muscle movements of facial expressions. The duration of the micro-expression is between 1/25 and 1/5 s, and only part of the muscle movements are occurring, although sometimes it contains all muscle movements of the macro expression. Considering the damaged ability of depressive people to recognize facial expressions, this study aimed to examine whether micro-expression could be detected by depressive patients and how the recognition performance of micro-expression differs from that of macro expressions. According to the previous results of depressive people on the recognition performance of macro expression and brief expression, we speculate that their ability to identify micro-expression is also likely to be lower than healthy people, and this difference may be amplified due to the increased difficulty of recognition. In this study, macro expression recognition test, brief affect recognition test and micro-facial recognition test were used to examine the performance difference in identifying macro expression, brief expression and micro-expression between depressive patients and healthy people. The results showed that depressive patients recognized expression and micro-expression worse than healthy people, and both depressive patients and healthy people performed worse in micro-expression recognition than that in the macro expression recognition. It also showed that the HAMD score was negative correlated with the accuracy of expression and micro-expression recognition. Previous studies have focused on the recognition of common facial expression (i.e., macro-expression) in depressive patients, and this study had extended this line work to micro-expression recognition. These results indicated that the expression recognition deficiency of depressive patients is independent from the duration of facial expression, reflecting the severity degree of depression to some extent.

RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition

Ensemble pruning is effective for improving the accuracy of expression recognition. This paper proposes a novel ensemble pruning algorithm called RTCRelief-F and applies it to facial expression recognition. RTCRelief-F uses a novel classifier-representation method that accounts for the interaction among classifiers and bases the classifier selection upon not only diversity but accuracy. Additionally, for the first time, RTCRelief-F, applies the Relief-F algorithm to evaluate the classifiers’ ability and resets the fusion order. Finally, the combination of a clustering-based ensemble pruning method and the ordering-based ensemble pruning method can both alleviate the dependence of a selected subset S on the adopted clustering strategies and guarantee the diversity of the selected subset S. The experimental results show that this method outperforms or competes with the original ensemble and some major state-of-the-art results on the data sets Fer2013, JAFFE, and CK $$+$$ .

Functionally distinct smiles elicit different physiological responses in an evaluative context

When people are being evaluated, their whole body responds. Verbal feedback causes robust activation in the hypothalamic-pituitary-adrenal (HPA) axis. What about nonverbal evaluative feedback? Recent discoveries about the social functions of facial expression have documented three morphologically distinct smiles, which serve the functions of reinforcement, social smoothing, and social challenge. In the present study, participants saw instances of one of three smile types from an evaluator during a modified social stress test. We find evidence in support of the claim that functionally different smiles are sufficient to augment or dampen HPA axis activity. We also find that responses to the meanings of smiles as evaluative feedback are more differentiated in individuals with higher baseline high-frequency heart rate variability (HF-HRV), which is associated with facial expression recognition accuracy. The differentiation is especially evident in response to smiles that are more ambiguous in context. Findings suggest that facial expressions have deep physiological implications and that smiles regulate the social world in a highly nuanced fashion.

A recursive framework for expression recognition: from web images to deep models to game dataset

In this paper, we propose a recursive framework to recognize facial expressions from images in real scenes. Unlike traditional approaches that typically focus on developing and refining algorithms for improving recognition performance on an existing dataset, we integrate three important components in a recursive manner: facial dataset generation, facial expression recognition model building, and interactive interfaces for testing and new data collection. To start with, we first create candid images for facial expression (CIFE) dataset. We then apply a convolutional neural network (CNN) to CIFE and build a CNN model for web image expression classification. In order to increase the expression recognition accuracy, we also fine-tune the CNN model and thus obtain a better CNN facial expression recognition model. Based on the fine-tuned CNN model, we design a facial expression game engine and collect a new and more balanced dataset, GaMo. The images of this dataset are collected from the different expressions our game users make when playing the game. Finally, we run yet another recursive step—a self-evaluation of the quality of the data labeling and propose a self-cleansing mechanism for improve the quality of the data. We evaluate the GaMo and CIFE datasets and show that our recursive framework can help build a better facial expression model for dealing with real scene facial expression tasks.

Survey on Different Methods to Improve Accuracy of The Facial Expression Recognition Using Artificial Neural Networks

Survey on Different Methods to Improve Accuracy of The Facial Expression Recognition Using Artificial Neural Networks

Local Directional Ternary Pattern for Facial Expression Recognition.

This paper presents a new face descriptor, local directional ternary pattern (LDTP), for facial expression recognition. LDTP efficiently encodes information of emotion-related features (ı.e., eyes, eyebrows, upper nose, and mouth) by using the directional information and ternary pattern in order to take advantage of the robustness of edge patterns in the edge region while overcoming weaknesses of edge-based methods in smooth regions. Our proposal, unlike existing histogram-based face description methods that divide the face into several regions and sample the codes uniformly, uses a two-level grid to construct the face descriptor while sampling expression-related information at different scales. We use a coarse grid for stable codes (highly related to non-expression), and a finer one for active codes (highly related to expression). This multi-level approach enables us to do a finer grain description of facial motions while still characterizing the coarse features of the expression. Moreover, we learn the active LDTP codes from the emotion-related facial regions. We tested our method by using person-dependent and independent cross-validation schemes to evaluate the performance. We show that our approaches improve the overall accuracy of facial expression recognition on six data sets.

Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition

Machine recognition of a handwritten mathematical expression (HME) is challenging due to the ambiguities of handwritten symbols and the two-dimensional structure of mathematical expressions. Inspired by recent work in deep learning, we present Watch, Attend and Parse (WAP), a novel end-to-end approach based on neural network that learns to recognize HMEs in a two-dimensional layout and outputs them as one-dimensional character sequences in LaTeX format. Inherently unlike traditional methods, our proposed model avoids problems that stem from symbol segmentation, and it does not require a predefined expression grammar. Meanwhile, the problems of symbol recognition and structural analysis are handled, respectively, using a watcher and a parser. We employ a convolutional neural network encoder that takes HME images as input as the watcher and employ a recurrent neural network decoder equipped with an attention mechanism as the parser to generate LaTeX sequences. Moreover, the correspondence between the input expressions and the output LaTeX sequences is learned automatically by the attention mechanism. We validate the proposed approach on a benchmark published by the CROHME international competition. Using the official training dataset, WAP significantly outperformed the state-of-the-art method with an expression recognition accuracy of 46.55% on CROHME 2014 and 44.55% on CROHME 2016.

Facial Expression Recognition Method Based on Stacked Denoising Autoencoders and Feature Reduction

Based on the deep learning theory, a novel facial expression recognition method, which utilizes both Principal Component Analysis (PCA) and stacked denoising autoencoders (SDAE), is proposed in this paper. At first, PCA is used as a linear dimensionality reduction method on the expression features, and subsequently non-linear dimensionality is further learnt by a greed layer-wise method of stacked denoising autoencoders. Then, some meaningful and low-dimensioned expression features can be learnt and used to classify. The comparative experiment results show that the proposed method is more effective than some other expression recognition methods based on deep learning theories and it can also get higher expression recognition accuracy than traditional non-deep learning based expression recognition methods.

Emotion Recognition in Nonverbal Face-to-Face Communication

The perception of emotional facial expressions may activate corresponding facial muscles in the receiver, also referred to as facial mimicry. Facial mimicry is highly dependent on the context and type of facial expressions. While previous research almost exclusively investigated mimicry in response to pictures or videos of emotional expressions, studies with a real, face-to-face partner are still rare. Here we compared facial mimicry of angry, happy, and sad expressions and emotion recognition in a dyadic face-to-face setting. In sender-receiver dyads, we recorded facial electromyograms in parallel. Senders communicated to the receivers—with facial expressions only—the emotions felt during specific personal situations in the past, eliciting anger, happiness, or sadness. Receivers mostly mimicked happiness, to a lesser degree, sadness, and anger as the least mimicked emotion. In actor-partner interdependence models we showed that the receivers’ own facial activity influenced their ratings, which increased the agreement between the senders’ and receivers’ ratings for happiness, but not for angry and sad expressions. These results are in line with the Emotion Mimicry in Context View, holding that humans mimic happy expressions according to affiliative intentions. The mimicry of sad expressions is less intense, presumably because it signals empathy and might imply personal costs. Direct anger expressions are mimicked the least, possibly because anger communicates threat and aggression. Taken together, we show that incidental facial mimicry in a face-to-face setting is positively related to the recognition accuracy for non-stereotype happy expressions, supporting the functionality of facial mimicry.

Use of 3D faces facilitates facial expression recognition in children

This study assessed whether presenting 3D face stimuli could facilitate children’s facial expression recognition. Seventy-one children aged between 3 and 6 participated in the study. Their task was to judge whether a face presented in each trial showed a happy or fearful expression. Half of the face stimuli were shown with 3D representations, whereas the other half of the images were shown as 2D pictures. We compared expression recognition under these conditions. The results showed that the use of 3D faces improved the speed of facial expression recognition in both boys and girls. Moreover, 3D faces improved boys’ recognition accuracy for fearful expressions. Since fear is the most difficult facial expression for children to recognize, the facilitation effect of 3D faces has important practical implications for children with difficulties in facial expression recognition. The potential benefits of 3D representation for other expressions also have implications for developing more realistic assessments of children’s expression recognition.

Identification of Emotional Facial Expressions: Effects of Expression, Intensity, and Sex on Eye Gaze.

The identification of emotional expressions is vital for social interaction, and can be affected by various factors, including the expressed emotion, the intensity of the expression, the sex of the face, and the gender of the observer. This study investigates how these factors affect the speed and accuracy of expression recognition, as well as dwell time on the two most significant areas of the face: the eyes and the mouth. Participants were asked to identify expressions from female and male faces displaying six expressions (anger, disgust, fear, happiness, sadness, and surprise), each with three levels of intensity (low, moderate, and normal). Overall, responses were fastest and most accurate for happy expressions, but slowest and least accurate for fearful expressions. More intense expressions were also classified most accurately. Reaction time showed a different pattern, with slowest response times recorded for expressions of moderate intensity. Overall, responses were slowest, but also most accurate, for female faces. Relative to male observers, women showed greater accuracy and speed when recognizing female expressions. Dwell time analyses revealed that attention to the eyes was about three times greater than on the mouth, with fearful eyes in particular attracting longer dwell times. The mouth region was attended to the most for fearful, angry, and disgusted expressions and least for surprise. These results extend upon previous findings to show important effects of expression, emotion intensity, and sex on expression recognition and gaze behaviour, and may have implications for understanding the ways in which emotion recognition abilities break down.

Neural substrates of the ability to recognize facial expressions: a voxel-based morphometry study.

The recognition of facial expressions of emotion is adaptive for human social interaction, but the ability to do this and the manner in which it is achieved differs among individuals. Previous functional neuroimaging studies have demonstrated that some brain regions, such as the inferior frontal gyrus (IFG), are active during the response to emotional facial expressions in healthy participants, and lesion studies have demonstrated that damage to these structures impairs the recognition of facial expressions. However, it remains to be established whether individual differences in the structure of these regions could be associated with differences in the ability to recognize facial expressions. We investigated this issue using acquired structural magnetic resonance imaging, and assessed the performance of healthy adults with respect to recognition of the facial expressions of six basic emotions. The gray matter volume of the right IFG positively correlated with the total accuracy of facial expression recognition. This suggests that individual differences in the ability to recognize facial expressions are associated with differences in the structure of the right IFG. Furthermore, the mirror neuron activity of the IFG may be important for establishing efficient facial mimicry to facilitate emotion recognition.

A differing bidirectional impact on the recognition accuracy of facial expression.

This study explored a bidirectional impact on the recognition accuracy of various facial expressions deriving from both the observer and sender in a sample of Chinese participants. A facial manipulation task was used to examine the ability of an observer's facial feedback to modulate the recognition of various facial expressions. Furthermore, the effect of a sender's facial expression with an open or closed mouth on recognition accuracy was investigated. The results showed that only recognition accuracy of a sad facial expression was influenced simultaneously by bidirectional sources from a sender and observer. Moreover, the impact of the unidirectional cue of a sender's facial feature (i.e., mouth openness) on happy and neutral faces was found to influence the recognition accuracy of these faces, but not the observer's bodily state. These findings indicate that the bidirectional impact derived from an observer and sender on facial expression recognition accuracy differs for emotional and neutral expressions.

Improved GA and Pareto optimization-based facial expression recognition

Purpose The purpose of this paper is to improve the accuracy of the facial expression recognition by using genetic algorithm (GA) with an appropriate fitness evaluation function and Pareto optimization model with two new objective functions. Design/methodology/approach To achieve facial expression recognition with high accuracy, the Haar-like features representation approach and the bilateral filter are first used to preprocess the facial image. Second, the uniform local Gabor binary patterns are used to extract the facial feature so as to reduce the feature dimension. Third, an improved GA and Pareto optimization approach are used to select the optimal significant features. Fourth, the random forest classifier is chosen to achieve the feature classification. Subsequently, some comparative experiments are implemented. Finally, the conclusion is drawn and some future research topics are pointed out. Findings The experiment results show that the proposed facial expression recognition algorithm outperforms ones in the existing literature in terms of both the actuary and computational time. Originality/value The GA and Pareto optimization algorithm are combined to select the optimal significant feature. To improve the accuracy of the facial expression recognition, the GA is improved by adjusting an appropriate fitness evaluation function, and a new Pareto optimization model is proposed that contains two objective functions indicating the achievements in minimizing within-class variations and in maximizing between-class variations.

Facial expression recognition through adaptive learning of local motion descriptor

A novel bag-of-words based approach is proposed for recognizing facial expressions corresponding to each of the six basic prototypic emotions from a video sequence. Each video sequence is represented as a specific combination of local (in spatio-temporal scale) motion patterns. These local motion patterns are captured in motion descriptors (MDs) which are unique combinations of optical flow and image gradient. These MDs can be compared to the words in the bag-of-words setting. Generally, the key-words in the wordbook as reported in the literature, are rigid, i.e., are taken as it is from the training data and cannot generalize well. We propose a novel adaptive learning technique for the key-words. The adapted key-MDs better represent the local motion patterns of the videos and generalize well to the unseen data and thus give better expression recognition accuracy. To test the efficiency of the proposed approach, we have experimented extensively on three well known datasets. We have also compared the results with existing state-of-the-art expression descriptors. Our method gives better accuracy. The proposed approach have been able to reduce the training time including the time for feature-extraction more than nine times and test time more than twice as compared to current state-of-the-art descriptor.

Dissociation between recognition and detection advantage for facial expressions: a meta-analysis.

Happy facial expressions are recognized faster and more accurately than other expressions in categorization tasks, whereas detection in visual search tasks is widely believed to be faster for angry than happy faces. We used meta-analytic techniques for resolving this categorization versus detection advantage discrepancy for positive versus negative facial expressions. Effect sizes were computed on the basis of the r statistic for a total of 34 recognition studies with 3,561 participants and 37 visual search studies with 2,455 participants, yielding a total of 41 effect sizes for recognition accuracy, 25 for recognition speed, and 125 for visual search speed. Random effects meta-analysis was conducted to estimate effect sizes at population level. For recognition tasks, an advantage in recognition accuracy and speed for happy expressions was found for all stimulus types. In contrast, for visual search tasks, moderator analysis revealed that a happy face detection advantage was restricted to photographic faces, whereas a clear angry face advantage was found for schematic and "smiley" faces. Robust detection advantage for nonhappy faces was observed even when stimulus emotionality was distorted by inversion or rearrangement of the facial features, suggesting that visual features primarily drive the search. We conclude that the recognition advantage for happy faces is a genuine phenomenon related to processing of facial expression category and affective valence. In contrast, detection advantages toward either happy (photographic stimuli) or nonhappy (schematic) faces is contingent on visual stimulus features rather than facial expression, and may not involve categorical or affective processing.