Facial Expression Information Research Articles

Facial Expression Recognition from Video Sequences Based on Spatial-Temporal Motion Local Binary Pattern and Gabor Multiorientation Fusion Histogram

This paper proposes novel framework for facial expressions analysis using dynamic and static information in video sequences. First, based on incremental formulation, discriminative deformable face alignment method is adapted to locate facial points to correct in-plane head rotation and break up facial region from background. Then, spatial-temporal motion local binary pattern (LBP) feature is extracted and integrated with Gabor multiorientation fusion histogram to give descriptors, which reflect static and dynamic texture information of facial expressions. Finally, a one-versus-one strategy based multiclass support vector machine (SVM) classifier is applied to classify facial expressions. Experiments on Cohn-Kanade (CK) + facial expression dataset illustrate that integrated framework outperforms methods using single descriptors. Compared with other state-of-the-art methods on CK+, MMI, and Oulu-CASIA VIS datasets, our proposed framework performs better.

A dynamic framework based on local Zernike moment and motion history image for facial expression recognition

A dynamic descriptor facilitates robust recognition of facial expressions in video sequences. The current two main approaches to the recognition are basic emotion recognition and recognition based on facial action coding system (FACS) action units. In this paper we focus on basic emotion recognition and propose a spatio-temporal feature based on local Zernike moment in the spatial domain using motion change frequency. We also design a dynamic feature comprising motion history image and entropy. To recognise a facial expression, a weighting strategy based on the latter feature and sub-division of the image frame is applied to the former to enhance the dynamic information of facial expression, and followed by the application of the classical support vector machine. Experiments on the CK+ and MMI datasets using leave-one-out cross validation scheme demonstrate that the integrated framework achieves a better performance than using individual descriptor separately. Compared with six state-of-arts methods, the proposed framework demonstrates a superior performance.

At a Single Glance: Fast Periodic Visual Stimulation Uncovers the Spatio-Temporal Dynamics of Brief Facial Expression Changes in the Human Brain.

Detecting brief changes of facial expression is vital for social communication. Yet, how reliably, how fast these changes are detected and how long they are processed in the human brain remain unknown. High-density electroencephalogram (EEG) was recorded in 18 participants presented with a neutral-expression face at a rate of 5.88 Hz (F) for 80 s. Every five faces, the face changed expression to fear, disgust or happiness (different stimulation sequences). The resulting 1.18 Hz (F/5) EEG response and its harmonics objectively indexed detection of a brief change of facial expression. This response was recorded in every participant in a few minutes but was largely reduced for inverted faces, indicating that it reflects high-level processes. Although this response focused on occipito-temporal sites, different expression changes evoked reliably distinct topographical maps, pointing to partly distinct neural generators. These effects were also observed at a faster 12 Hz frequency rate and a lower ratio of expression change (1/9). Time-domain analysis showed that a brief change of expression inserted in a dynamic stimulation sequence elicits specific occipito-temporal responses between 100 and 310 ms, indicating a rapid change detection process followed by a long integration period of facial expression information in the human brain.

The integration of visual context information in facial emotion recognition in 5- to 15-year-olds

The current study investigated the role of congruent visual context information in the recognition of facial emotional expression in 190 participants from 5 to 15years of age. Children performed a matching task that presented pictures with different facial emotional expressions (anger, disgust, happiness, fear, and sadness) in two conditions: with and without a visual context. The results showed that emotions presented with visual context information were recognized more accurately than those presented in the absence of visual context. The context effect remained steady with age but varied according to the emotion presented and the gender of participants. The findings demonstrated for the first time that children from the age of 5years are able to integrate facial expression and visual context information, and this integration improves facial emotion recognition.

ERP comparison study of face gender and expression processing in unattended condition

This paper explores the differences within the underlying brain mechanism for facial expression and gender information processing. The study recorded Event-Related Potentials (ERPs) of participants while they were performing a cross-change detection task, in which the cross was peripherally surrounded by four facial stimuli. For investigating facial expression processing, either four faces with positive expressions from two females and two males were presented infrequently among four faces with negative expressions, or four faces with negative expressions from two females and two males were presented infrequently among four faces with positive expressions. For gender information processing, four female faces were presented infrequently among four male faces, or vice versa.The findings showed that the latency of facial emotion-related visual mismatch negativity (vMMN) was shorter than that of facial gender-related vMMN, and that their related brain regions presented some differences. The results can be viewed as evidence of the difference by which the human brain processes facial emotion and gender information.

A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences

Facial expression causes different parts of the facial region to change over time and thus dynamic descriptors are inherently more suitable than static descriptors for recognising facial expressions. In this paper, we extend the spatial pyramid histogram of gradients to spatio-temporal domain to give 3-dimensional facial features and integrate them with dense optical flow to give a spatio-temporal descriptor which extracts both the spatial and dynamic motion information of facial expressions. A multi-class support vector machine based classifier with one-to-one strategy is used to recognise facial expressions. Experiments on the CK+ and MMI datasets using leave-one-out cross validation scheme demonstrate that the integrated framework achieves a better performance than using individual descriptor separately. Compared with six state of the art methods, the proposed framework demonstrates a superior performance.

Recognizing age-separated face images: humans and machines.

Humans utilize facial appearance, gender, expression, aging pattern, and other ancillary information to recognize individuals. It is interesting to observe how humans perceive facial age. Analyzing these properties can help in understanding the phenomenon of facial aging and incorporating the findings can help in designing effective algorithms. Such a study has two components - facial age estimation and age-separated face recognition. Age estimation involves predicting the age of an individual given his/her facial image. On the other hand, age-separated face recognition consists of recognizing an individual given his/her age-separated images. In this research, we investigate which facial cues are utilized by humans for estimating the age of people belonging to various age groups along with analyzing the effect of one's gender, age, and ethnicity on age estimation skills. We also analyze how various facial regions such as binocular and mouth regions influence age estimation and recognition capabilities. Finally, we propose an age-invariant face recognition algorithm that incorporates the knowledge learned from these observations. Key observations of our research are: (1) the age group of newborns and toddlers is easiest to estimate, (2) gender and ethnicity do not affect the judgment of age group estimation, (3) face as a global feature, is essential to achieve good performance in age-separated face recognition, and (4) the proposed algorithm yields improved recognition performance compared to existing algorithms and also outperforms a commercial system in the young image as probe scenario.

Interference between conscious and unconscious facial expression information.

There is ample evidence to show that many types of visual information, including emotional information, could be processed in the absence of visual awareness. For example, it has been shown that masked subliminal facial expressions can induce priming and adaptation effects. However, stimulus made invisible in different ways could be processed to different extent and have differential effects. In this study, we adopted a flanker type behavioral method to investigate whether a flanker rendered invisible through Continuous Flash Suppression (CFS) could induce a congruency effect on the discrimination of a visible target. Specifically, during the experiment, participants judged the expression (either happy or fearful) of a visible face in the presence of a nearby invisible face (with happy or fearful expression). Results show that participants were slower and less accurate in discriminating the expression of the visible face when the expression of the invisible flanker face was incongruent. Thus, facial expression information rendered invisible with CFS and presented a different spatial location could enhance or interfere with consciously processed facial expression information.

Quantifying human sensitivity to spatio-temporal information in dynamic faces.

A great deal of perceptual and social information is conveyed by facial motion. Here, we investigated observers' sensitivity to the complex spatio-temporal information in facial expressions and what cues they use to judge the similarity of these movements. We motion-captured four facial expressions and decomposed them into time courses of semantically meaningful local facial actions (e.g., eyebrow raise). We then generated approximations of the time courses which differed in the amount of information about the natural facial motion they contained, and used these and the original time courses to animate an avatar head. Observers chose which of two animations based on approximations was more similar to the animation based on the original time course. We found that observers preferred animations containing more information about the natural facial motion dynamics. To explain observers' similarity judgments, we developed and used several measures of objective stimulus similarity. The time course of facial actions (e.g., onset and peak of eyebrow raise) explained observers' behavioral choices better than image-based measures (e.g., optic flow). Our results thus revealed observers' sensitivity to changes of natural facial dynamics. Importantly, our method allows a quantitative explanation of the perceived similarity of dynamic facial expressions, which suggests that sparse but meaningful spatio-temporal cues are used to process facial motion.

Network Interactions Explain Sensitivity to Dynamic Faces in the Superior Temporal Sulcus.

The superior temporal sulcus (STS) in the human and monkey is sensitive to the motion of complex forms such as facial and bodily actions. We used functional magnetic resonance imaging (fMRI) to explore network-level explanations for how the form and motion information in dynamic facial expressions might be combined in the human STS. Ventral occipitotemporal areas selective for facial form were localized in occipital and fusiform face areas (OFA and FFA), and motion sensitivity was localized in the more dorsal temporal area V5. We then tested various connectivity models that modeled communication between the ventral form and dorsal motion pathways. We show that facial form information modulated transmission of motion information from V5 to the STS, and that this face-selective modulation likely originated in OFA. This finding shows that form-selective motion sensitivity in the STS can be explained in terms of modulation of gain control on information flow in the motion pathway, and provides a substantial constraint for theories of the perception of faces and biological motion.

Facial Expression Recognition System Using Facial Characteristic Points And ID3

Facial expression is one of the most powerful, natural, and abrupt means for human beings which have the knack to communicate emotion and regulate inter-personal behaviour. In this paper we present a novel approach for facial expression detection using decision tree. Facial expression information is mostly concentrate on facial expression information regions, so the mouth, eye and eyebrow regions are segmented from the facial expression images firstly. Using these templates we calculate 30 facial characteristics points (FCP’s). These facial characteristic points describe the position and shape of the above three organs to find diverse parameters which are input to the decision tree for recognizing different facial expressions.

Development of Computer Vision System to Extract Kansei Information from Facial Expressions

Kansei engineering is one field of engineering where the feelings of humans are applied to manufacturing. If Kansei can be extracted, the design, development, and evaluation of various things are possible. This study aims to build an automatic design and evaluation system for agricultural products and landscapes and then utilize the system for the evaluation and improvement of educational effects. In this presentation, we built a low price and near real-time system to extract Kansei information from the facial expressions of users and obtained knowledge about Kansei. Additionally, we created an algorithm to discriminate Kansei. We built a system to obtain facial expression information of a user from a camera image as our experimental equipment. The characteristic values calculated from them were the variation (or the normalized polygon area change ratio) of the four items; the inner ends of the eyebrows, upper parts of the eyes, lower parts of the eyes, and the corners of the mouth. The target Kansei was three items: positive Kansei (liking, fun, and happy) and negative Kansei (unpleasantness and hatred). The results are as follows:First, for the target three Kansei, changes could be captured through the four specified normalized polygon area change rates. As for the positive Kansei, the changes in the corners of the mouth were greater than those of the others in particular. As for the negative Kansei, the changes in the lower parts of the eyes were slightly greater than those of the others and, as for the Kansei of surprise, the changes in the corners of the mouth were slightly greater. The direction of the change showed a tendency where the positive Kansei increased at the corners of the mouth in particular. The negative Kansei showed a strong tendency to decrease at the inner ends of the eyebrows and the lower parts of the eyes in particular. The Kansei of surprise showed a tendency where the normalized polygon area change rates increased as a whole. Next, we selected one subject because the relationship between the facial expressions and Kansei varies among individuals, and then we built an algorithm to discriminate the three Kansei features. As for the Kansei of positive – negative and negative – surprise, it became possible to discriminate at a rate of nearly 100% from the information of the corners of the mouth. It became possible to discriminate the positive-surprise at a rate of about 80% from the information of the upper parts of the eyes. As described above, although it is necessary to prepare the template image of the seven points of the target person and adjust the parameters after measuring the Kansei facial expression in advance, a low price and near real-time computer vision system to extract facial expression Kansei could be built.

Faces in context: Modulation of expression processing by situational information

Numerous studies using the event-related potential (ERP) technique have found that emotional expressions modulate ERP components appearing at different post-stimulus onset times and are indicative of different stages of face processing. With the aim of studying the time course of integration of context and facial expression information, we investigated whether these modulations are sensitive to the situational context in which emotional expressions are perceived. Participants were asked to identify the expression of target faces that were presented immediately after reading short sentences that described happy or anger-inducing situations. The main manipulation was the congruency between the emotional content of the sentences and the target expression. Context-independent amplitude modulation of the N170 and N400 components by emotional expression was observed. On the other hand, context effects appeared on a later component (late positive potential, or LPP), with enhanced amplitudes on incongruent trials. These results show that the early stages of face processing where emotional expressions are coded are not sensitive to verbal information about the situation in which they appear. The timing of context congruency effects suggests that integration of facial expression with situational information occurs at a later stage, probably related to the detection of affective congruency.

Processing of ambiguous emotional messages in brain injured patients with and without subcortical lesions

Background: Emotional information in communicative messages is conveyed through a combination of linguistic and paralinguistic cues. Listeners must simultaneously process affective prosody, facial expressions, body gestures, and semantic information in order to determine the speaker's emotional intent. Findings from recent research support the involvement of subcortical structures in unimodal emotion recognition; however, given the propensity for emotional information to be conveyed across multiple modalities simultaneously, clinical advances require an understanding of how subcortical and cortical structures are involved in the recognition of multi-modal stimuli. Aims: To determine if individuals with and without a history of brain damage demonstrate affective processing biases on ambiguous and congruous multi-modal emotive stimuli.1 Methods & Procedures: Twenty individuals with brain damage and five individuals without a history of brain injury (NC) were included in the study. Participants with brain damage were grouped by depth and location of lesion site (left cortical [LC], right cortical [RC], left subcortical-cortical [LS], and right subcortical-cortical [RS]) to determine the effects of damage location on emotion processing. Participants identified emotions from stimuli containing incongruent affective information across two tasks using combinations of verbal, prosodic, and visual (i.e. facial expression) information. In Task 1, affectively charged sentences were presented with conflicting paralinguistic information (speech prosody, facial expression, and a combined modality condition). Task 2 involved the presentation of conflicting facial expression and prosodic information in linguistically neutral sentences. Response preferences for specific modalities were examined to understand potential processing biases. Outcomes & Results: The NC and LC groups demonstrated a bias for selecting emotions displayed by paralinguistic information in all Task 1 conditions. The RC group exhibited a similar preference when facial expression was included in the stimulus, but a decreased bias for selecting paralinguistic information when speech prosody was presented alone. In contrast to the NC and cortical damage groups, the RS group consistently selected emotions based on linguistic cues. The LS group exhibited the greatest inter-subject variability. Group patterns suggested no clear preference for either linguistic or paralinguistic information when speech prosody was included in the stimulus. In the task involving conflicting facial expression and linguistic information, the LS group exhibited a bias for facial expression information. All groups demonstrated a preference for facial expression information over speech prosody; however, the RS group's preference was significantly weaker. Conclusions: The data suggest that combined cortical and subcortical damage disrupts typical processing strategies for emotion recognition.

Recognition of Facial Expressions and Prosodic Cues with Graded Emotional Intensities in Adults with Asperger Syndrome

This study investigated the ability of adults with Asperger syndrome to recognize emotional categories of facial expressions and emotional prosodies with graded emotional intensities. The individuals with Asperger syndrome showed poorer recognition performance for angry and sad expressions from both facial and vocal information. The group difference in facial expression recognition was prominent for stimuli with low or intermediate emotional intensities. In contrast to this, the individuals with Asperger syndrome exhibited lower recognition accuracy than typically-developed controls mainly for emotional prosody with high emotional intensity. In facial expression recognition, Asperger and control groups showed an inversion effect for all categories. The magnitude of this effect was less in the Asperger group for angry and sad expressions, presumably attributable to reduced recruitment of the configural mode of face processing. The individuals with Asperger syndrome outperformed the control participants in recognizing inverted sad expressions, indicating enhanced processing of local facial information representing sad emotion. These results suggest that the adults with Asperger syndrome rely on modality-specific strategies in emotion recognition from facial expression and prosodic information.

Robust Automatic Facial Expression Detection Method

Recently, the recognition of occluded facial expressions attract more and more people’s attention . Sparse representation based classification (SRC) method gives good performance on face recognition (FR) and facial expression recognition (FER), well-known for its robustness to occlusion. Histograms of Oriented Gradient (HOG) descriptors are very efficient to represent the shape information of different facial expressions and robust to various illumination. Since, this paper propos es a novel method by using HOG descriptors conjunction with SRC framework for FER. Experiment results show that the proposed method gives better performance than the existing state-of-the-art methods. Furthermore, the proposed method is not only robust to assigned occlusion s , but also to random occlusion s .

Towards Efficient Multi-Modal Emotion Recognition

The paper presents a multi-modal emotion recognition system exploiting audio and video (i.e., facial expression) information. The system first processes both sources of information individually to produce corresponding matching scores and then combines the computed matching scores to obtain a classification decision. For the video part of the system, a novel approach to emotion recognition, relying on image-set matching, is developed. The proposed approach avoids the need for detecting and tracking specific facial landmarks throughout the given video sequence, which represents a common source of error in video-based emotion recognition systems, and, therefore, adds robustness to the video processing chain. The audio part of the system, on the other hand, relies on utterance-specific Gaussian Mixture Models (GMMs) adapted from a Universal Background Model (UBM) via the maximum a posteriori probability (MAP) estimation. It improves upon the standard UBM-MAP procedure by exploiting gender information when building the utterance-specific GMMs, thus ensuring enhanced emotion recognition performance. Both the uni-modal parts as well as the combined system are assessed on the challenging multi-modal eNTERFACE'05 corpus with highly encouraging results. The developed system represents a feasible solution to emotion recognition that can easily be integrated into various systems, such as humanoid robots, smart surveillance systems and alike.

Dynamic and Static Facial Expressions Decoded from Motion-Sensitive Areas in the Macaque Monkey

Humans adeptly use visual motion to recognize socially relevant facial information. The macaque provides a model visual system for studying neural coding of expression movements, as its superior temporal sulcus (STS) possesses brain areas selective for faces and areas sensitive to visual motion. We used functional magnetic resonance imaging and facial stimuli to localize motion-sensitive areas [motion in faces (Mf) areas], which responded more to dynamic faces compared with static faces, and face-selective areas, which responded selectively to faces compared with objects and places. Using multivariate analysis, we found that information about both dynamic and static facial expressions could be robustly decoded from Mf areas. By contrast, face-selective areas exhibited relatively less facial expression information. Classifiers trained with expressions from one motion type (dynamic or static) showed poor generalization to the other motion type, suggesting that Mf areas employ separate and nonconfusable neural codes for dynamic and static presentations of the same expressions. We also show that some of the motion sensitivity elicited by facial stimuli was not specific to faces but could also be elicited by moving dots, particularly in fundus of the superior temporal and middle superior temporal polysensory/lower superior temporal areas, confirming their already well established low-level motion sensitivity. A different pattern was found in anterior STS, which responded more to dynamic than to static faces but was not sensitive to dot motion. Overall, we show that emotional expressions are mostly represented outside of face-selective cortex, in areas sensitive to motion. These regions may play a fundamental role in enhancing recognition of facial expression despite the complex stimulus changes associated with motion.

Gauss–Laguerre wavelet textural feature fusion with geometrical information for facial expression identification

Facial expressions are a valuable source of information that accompanies facial biometrics. Early detection of physiological and psycho-emotional data from facial expressions is linked to the situational awareness module of any advanced biometric system for personal state re/identification. In this article, a new method that utilizes both texture and geometric information of facial fiducial points is presented. We investigate Gauss–Laguerre wavelets, which have rich frequency extraction capabilities, to extract texture information of various facial expressions. Rotation invariance and the multiscale approach of these wavelets make the feature extraction robust. Moreover, geometric positions of fiducial points provide valuable information for upper/lower face action units. The combination of these two types of features is used for facial expression classification. The performance of this system has been validated on three public databases: the JAFFE, the Cohn-Kanade, and the MMI image.

Emotion Detection System Based on Speech and Facial Signals

This paper introduces the present status of speech emotion detection. In order to improve the emotion recognition rate of single mode, the bimodal fusion method based on speech and facial expression is proposed. First, we establishes emotional database include speech and facial expression. For different emotions, calm, happy, surprise, anger, sad, we extract ten speech parameters and use the PCA method to detect the speech emotion. Then we analyze the bimodal emotion detection of fusing facial expression information. The experiment results show that the emotion recognition rate with bimodal fusion is about 6 percent points higher than the recognition rate with only speech prosodic features