Facial Expression Classification Research Articles

Facial expression recognition using best tree RD-LGP encoded features and HMM

Hand and face gestures enable deaf people to communicate in their daily lives rather than speaking. This paper describes a hidden Markov model (HMM)-based framework for face sign recognition and detection. The observation vectors used to characterize the states of the HMM are obtained using the best tree local gradient pattern (LGP) encoded features. Each face gesture is modeled as a five-state HMM. The problem of facial expression classification is posed as a composite seven-classes multi-hypothesis Bayesian test. The likelihood ratio test showed that the overall recognition rate for the proposed model is higher than the HMM-local binary pattern descriptor by 6.4%. The overall recognition rate is enhanced by 8.6% using the discrete wavelet packet best tree decomposition filter as a pre-processing noise removal tool. In addition, the overall recognition rate ranges from 84.3%, for the seven classes Bayesian test, to 100%, for lower number of classes depending upon the type of the face gesture. The proposed face expression algorithm reduces significantly the computational complexity of previous HMM-based face expression recognition systems, and still preserve the recognition rate.

Constructing emotion categorization: Insights from developmental psychology applied to a young adult sample.

Previous research has found that the categorization of emotional facial expressions is influenced by a variety of factors, such as processing time, facial mimicry, emotion labels, and perceptual cues. However, past research has frequently confounded these factors, making it impossible to ascertain how adults use this varied information to categorize emotions. The current study is the first to explore the magnitude of impact for each of these factors on emotion categorization in the same paradigm. Participants (N = 102) categorized anger and disgust emotional facial expressions in a novel computerized task, modeled on similar tasks in the developmental literature with preverbal infants. Experimental conditions manipulated (a) whether the task was time-restricted, and (b) whether the labels "anger" and "disgust" were used in the instructions. Participants were significantly more accurate when provided with unlimited response time and emotion labels. Participants who were given restricted sorting time (2s) and no emotion labels tended to focus on perceptual features of the faces when categorizing the emotions, which led to low sorting accuracy. In addition, facial mimicry related to greater sorting accuracy. These results suggest that when high-level (labeling) categorization strategies are unavailable, adults use low-level (perceptual) strategies to categorize facial expressions. Methodological implications for the study of emotion are discussed. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Implementation and Comparison of Facial Expression Detection and Classification Techniques

Implementation and Comparison of Facial Expression Detection and Classification Techniques

Efficacy of theatre activities in facial expression categorization in schizophrenia

The aim of the study was to assess the efficacy of theatre activities in improving facial expression categorization in people who have schizophrenia by means of a controlled psychophysical experiment performed on matched groups of participants, and appropriate statistical analyses on collected data. In two experiments participants engaged in four alternative labelling tasks of pictures randomly taken from four morphed continua between two emotional facial expressions. Results of Experiment 1 showed that the identification rate was specifically lower for partipants who have schizophrenia-spectrum disorders (SSD) than for matched controls for the response fear, and less dramatically lower for the response sadness. Results of Experiment 2 showed that the specific impairment in the identification of fear expression improved in people with SSD who have experience of theatre activities with respect to matched SSD patients without theatre experience. These results suggest that theatre activities may be included in broad-based social cognitive training programs for those diagnosed with schizophrenia, and that it may be considered a useful means to directly affect social cognition, given the effects we found in the domain of facial expression recognition.

An objective signature of emotional expressions and context integration within a single glance: evidence from electroencephalographic frequency-tagging

The ability to quickly and accurately extract someone's emotional state from their face is crucial for social interaction. Over the last decades, the processing of emotional expressions has been studied mainly using isolated faces. However, at the behavioral level, contextual information often leads to radical changes in the categorization of facial expressions, yet the underlying mechanisms are not well understood (Aviezer et al., 2017, Current Opinion in Psychology, 17, 47–54; Barrett et al., 2011, Current Directions in Psychological Science, 20, 286 –290). Here we examined the impact of emotional visual scenes on the perception of emotional expressions within a single glance by means of fast periodic visual stimulation (FPVS). We recorded 128-channel EEG while participants viewed 60s sequences with a dual frequency-tagging paradigm (Boremanse et al, 2013, Journal of Vision (11):6, 1-18). We presented faces and scenes simultaneously, with each stimulus set flickering at specific frequency (f1=4.61 Hz and f2=5.99 Hz; frequencies were counterbalanced across stimuli). Each sequence displayed different faces with the same emotional expression (disgust, fear, or joy) within either positive or negative valence visual scenes. Periodic EEG responses at the image presentation frequencies (4.61 Hz and 5.99 Hz) captured general visual processing of the emotional faces and scenes, while intermodulation components (e.g. f2-f1: 5.99 – 4.61 Hz = 1.38 Hz) captured the integration between the emotional expressions and their context. At the group-level, emotional expressions elicited right-lateralized occipito-temporal electrophysiological responses that were stronger for negative valence expressions (especially disgust). Similarly, negative scenes elicited stronger neural responses than positive scenes over the medial occipital region. Finally, and critically, we observed intermodulation components that were prominent over right occipito-temporal sites and showed increased response amplitude for negative scenes, thereby providing an objective demonstration of the perceptual integration of emotional facial expressions with their emotional context.

Spatial frequencies for accurate categorization and discrimination of facial expressions

Spatial frequencies for accurate categorization and discrimination of facial expressions

Orientation information in encoding facial expressions

Previous research showed that we use different regions of a face to categorize different facial expressions, e.g. mouth region for identifying happy faces; eyebrows, eyes and upper part of nose for identifying angry faces. These findings imply that the spatial information along or close to the horizontal orientation might be more useful than others for facial expression recognition. In this study, we examined how the performance for recognizing facial expression depends on the spatial information along different orientations, and whether the pixel-level differences in the face images could account for subjects’ performance. Four facial expressions—angry, fearful, happy and sad—were tested. An orientation filter (bandwidth = 23°) was applied to restrict information within the face images, with the center of the filter ranged from 0° (horizontal) to 150° in steps of 30°. Accuracy for recognizing facial expression was measured for an unfiltered and the six filtered conditions. For all four facial expressions, recognition performance (normalized d′) was virtually identical for filter orientations of −30°, horizontal and 30°, and declined systematically as the filter orientation approached vertical. The information contained in mouth and eye regions is a significant predictor for subject’s response (based on the confusion patterns). We conclude that young adults with normal vision categorizes facial expression most effectively based on the spatial information around the horizontal orientation which captures primary changes of facial features across expressions. Across all spatial orientations, the information contained in mouth and eye regions contributes significantly to facial expression categorization.

Deep learning for the recognition of facial expression in the Colombian sign language

The communication of the deaf person has been considered as a set of manual movements, however many have ignored the importance of facial expression in the different communicative manifestations of this population; this is the reason because a group of Phonoaudiology, physiotherapy and engineering researchers integrate their knowledge to perform image processing and classification of facial expressions used in the Colombian Sign Language (LSC) analysis. The Objective was to establish the processing of images of facial expressions as a complementary means to manual movements for the interpretation of Colombian sign language. Qualitative study, descriptive, non-experimental design, for which four phases were considered, in the first, data collection was carried out through recordings of deaf people as linguistic models producing facial expressions corresponding to the vocabulary of the clinical scenario. In phase two, the images are processed to identify the characteristic patterns of each sign. In the third phase, two Deep Learning techniques are used to classify the captured gestures. In the fourth phase, the accuracy of the images was validated techniques used. For the classification process, six facial expressions corresponding to the words pain, inflammation, fracture, irritable colon, dizziness, diabetes were analyzed. In this process, two Deep Learning techniques were validated, obtaining that the Single Shot Multibox Detector SSD technique has a precision of 94, 2% compared to the Convolutional Neural Network (CNN) technique, which has a degree of accuracy of 89.05%. The development of technologies of this nature allows analyzing the facial expression in the communication of the deaf person as a distinctive feature for interaction with others. The algorithms of artificial vision based on Deep learning present a high level of efficiency in the classification of facial expression, this being an important factor to generate tools that facilitate the communication of deaf and hearing.

Thermal Augmented Expression Recognition.

Visible facial images provide geometric and appearance patterns of facial expressions and are sensitive to illumination changes. Thermal facial images record facial temperature distribution and are robust to light conditions. Therefore, expression recognition is enhanced by visible and thermal image fusion. In most cases, only visible images are available due to the widespread popularity of visible cameras and the high cost of thermal cameras. Thus, we propose a novel visible expression recognition method by using thermal infrared (IR) data as privileged information, which is only available during training. Specifically, we first learn a deep model for visible images and thermal images. Then we use the learned feature representations to train support vector machine (SVM) classifiers for expression classification. We jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Thermal IR images during training are then exploited to construct better facial representations and expression classifiers from visible images. We extend the proposed thermal augmented expression recognition method for partially unpaired data, acknowledging that visible images and thermal images maybe not be recorded synchronously. Experimental resulton the MAHNOB laughter database demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal IR images' supplementary role for visible facial expression recognition during training to obtain better facial representations and a better visible expression classifier. The proposed thermal augmented expression recognition method achieves state-of-the-art expression recognition performance for both paired and unpaired facial images.

Facial expression classification using salient pattern driven integrated geometric and textual features

Facial expression classification aims to recognize human emotion via face images. The major challenge of facial expression classification is how to extract discriminative features from the human face images to differentiate various emotions. To tackle this challenge, a new feature extraction approach is proposed in this paper. The proposed approach defines a new set of salient patterns at the facial keypoint locations. This is in contrast to conventional approaches that either represent the whole face image as a regular grid or use local image patches centered at all facial key point locations. Driven by the proposed salient patterns, both the geometric and textual features are extracted and then concatenated and further incorporated into a machine learning framework to perform facial expression classification. The proposed approach is evaluated in the well-known CK + benchmark dataset to demonstrate its superior performance.

Multivariate Pattern Classification of Facial Expressions Based on Large-Scale Functional Connectivity.

It is an important question how human beings achieve efficient recognition of others’ facial expressions in cognitive neuroscience, and it has been identified that specific cortical regions show preferential activation to facial expressions in previous studies. However, the potential contributions of the connectivity patterns in the processing of facial expressions remained unclear. The present functional magnetic resonance imaging (fMRI) study explored whether facial expressions could be decoded from the functional connectivity (FC) patterns using multivariate pattern analysis combined with machine learning algorithms (fcMVPA). We employed a block design experiment and collected neural activities while participants viewed facial expressions of six basic emotions (anger, disgust, fear, joy, sadness, and surprise). Both static and dynamic expression stimuli were included in our study. A behavioral experiment after scanning confirmed the validity of the facial stimuli presented during the fMRI experiment with classification accuracies and emotional intensities. We obtained whole-brain FC patterns for each facial expression and found that both static and dynamic facial expressions could be successfully decoded from the FC patterns. Moreover, we identified the expression-discriminative networks for the static and dynamic facial expressions, which span beyond the conventional face-selective areas. Overall, these results reveal that large-scale FC patterns may also contain rich expression information to accurately decode facial expressions, suggesting a novel mechanism, which includes general interactions between distributed brain regions, and that contributes to the human facial expression recognition.

IFER: facial expression recognition using automatically selected geometric eye and eyebrow features

Facial expressions have an important role in interpersonal communications and estimation of emotional states or intentions. Automatic recognition of facial expressions has led to many practical applications and became one of the important topics in computer vision. We present a facial expression recognition system that relies on geometry-based features extracted from eye and eyebrow regions of the face. The proposed system detects keypoints on frontal face images and forms a feature set using geometric relationships among groups of detected keypoints. Obtained feature set is refined and reduced using the sequential forward selection (SFS) algorithm and fed to a support vector machine classifier to recognize five facial expression classes. The proposed system, iFER (eye–eyebrow only facial expression recognition), is robust to lower face occlusions that may be caused by beards, mustaches, scarves, etc. and lower face motion during speech production. Preliminary experiments on benchmark datasets produced promising results outperforming previous facial expression recognition studies using partial face features, and comparable results to studies using whole face information, only slightly lower by ∼ 2.5 % compared to the best whole face facial recognition system while using only ∼ 1 / 3 of the facial region.

Knowing how you are feeling depends on what's on my mind: Cognitive load and expression categorization.

The ability to correctly interpret facial expressions is key to effective social interactions. People are well rehearsed and generally very efficient at correctly categorizing expressions. However, does their ability to do so depend on how cognitively loaded they are at the time? Using repeated-measures designs, we assessed the sensitivity of facial expression categorization to cognitive resources availability by measuring people's expression categorization performance during concurrent low and high cognitive load situations. In Experiment1, participants categorized the 6 basic upright facial expressions in a 6-automated-facial-coding response paradigm while maintaining low or high loading information in working memory (N = 40; 60 observations per load condition). In Experiment 2, they did so for both upright and inverted faces (N = 46; 60 observations per load and inversion condition). In both experiments, expression categorization for upright faces was worse during high versus low load. Categorization rates actually improved with increased load for the inverted faces. The opposing effects of cognitive load on upright and inverted expressions are explained in terms of a cognitive load-related dispersion in the attentional window. Overall, the findings support that expression categorization is sensitive to cognitive resources availability and moreover suggest that, in this paradigm, it is the perceptual processing stage of expression categorization that is affected by cognitive load. (PsycINFO Database Record

Positive Classification Advantage: Tracing the Time Course Based on Brain Oscillation.

The present study aimed to explore the modulation of frequency bands (alpha, beta, theta) underlying the positive facial expressions classification advantage within different post-stimulus time intervals (100–200 ms, 200–300 ms, 300–400 ms). For this purpose, we recorded electroencephalogram (EEG) activity during an emotion discrimination task for happy, sad and neutral faces. The correlation between the non-phase-locked power of frequency bands and reaction times (RTs) was assessed. The results revealed that beta played a major role in positive classification advantage (PCA) within the 100–200 and 300–400 ms intervals, whereas theta was important within the 200–300 ms interval. We propose that the beta band modulated the neutral and emotional face classification process, and that the theta band modulated for happy and sad face classification.

Classification of Facial Expressions from EEG signals using Wavelet Packet Transform and SVM for Wheelchair Control Operations

This paper describes the classification of facial expressions using EEG data. The entire procedure aims at controlling an electric wheelchair with a Brain Computer Interface (BCI) headset. The goal is to help the people who are suffering from locked-in syndrome to move or to pass the necessary signals. The headset consists of the electroencephalogram (EEG) cap comprising 16 electrodes attached to the amplifier out of which 14 electrodes are used for data acquisition while two are used as reference and ground. The EEG cap is placed on the head of the subject and various expressions (blink, eyebrows raise, smile, etc.) are performed on the subject. Muscle activities due to facial expressions can be observed from the recorded EEG signals. Expressions are classified and necessary signals are generated. The features are extracted using the wavelet packet transform processing method and classified primarily using Support Vector Machine (SVM).

Facial Expressions in Context: Electrophysiological Correlates of the Emotional Congruency of Facial Expressions and Background Scenes.

Facial expressions can display personal emotions and indicate an individual’s intentions within a social situation. They are extremely important to the social interaction of individuals. Background scenes in which faces are perceived provide important contextual information for facial expression processing. The purpose of this study was to explore the time course of emotional congruency effects in processing faces and scenes simultaneously by recording event-related potentials (ERPs). The behavioral results found that the categorization of facial expression was faster and more accurate when the face was emotionally congruent than incongruent with the emotion displayed by the scene. In ERPs the late positive potential (LPP) amplitudes were modulated by the emotional congruency between faces and scenes. Specifically, happy faces elicited larger LPP amplitudes within positive than within negative scenes and fearful faces within negative scenes elicited larger LPP amplitudes than within positive scenes. The results did not find the scene effects on the P1 and N170 components. These findings indicate that emotional congruency effects could occur in late stages of facial expression processing, reflecting motivated attention allocation.

Orientations for the successful categorization of facial expressions and their link with facial features.

Horizontal information was recently suggested to be crucial for face identification. In the present paper, we expand on this finding and investigate the role of orientations for all the basic facial expressions and neutrality. To this end, we developed orientation bubbles to quantify utilization of the orientation spectrum by the visual system in a facial expression categorization task. We first validated the procedure in Experiment 1 with a simple plaid-detection task. In Experiment 2, we used orientation bubbles to reveal the diagnostic-i.e., task relevant-orientations for the basic facial expressions and neutrality. Overall, we found that horizontal information was highly diagnostic for expressions-surprise excepted. We also found that utilization of horizontal information strongly predicted performance level in this task. Despite the recent surge of research on horizontals, the link with local features remains unexplored. We were thus also interested in investigating this link. In Experiment 3, location bubbles were used to reveal the diagnostic features for the basic facial expressions. Crucially, Experiments 2 and 3 were run in parallel on the same participants, in an interleaved fashion. This way, we were able to correlate individual orientation and local diagnostic profiles. Our results indicate that individual differences in horizontal tuning are best predicted by utilization of the eyes.

Spatially generalizable representations of facial expressions: Decoding across partial face samples

A network of cortical and sub-cortical regions is known to be important in the processing of facial expression. However, to date no study has investigated whether representations of facial expressions present in this network permit generalization across independent samples of face information (e.g., eye region vs mouth region). We presented participants with partial face samples of five expression categories in a rapid event-related fMRI experiment. We reveal a network of face-sensitive regions that contain information about facial expression categories regardless of which part of the face is presented. We further reveal that the neural information present in a subset of these regions: dorsal prefrontal cortex (dPFC), superior temporal sulcus (STS), lateral occipital and ventral temporal cortex, and even early visual cortex, enables reliable generalization across independent visual inputs (faces depicting the ‘eyes only’ vs ‘eyes removed’). Furthermore, classification performance was correlated to behavioral performance in STS and dPFC. Our results demonstrate that both higher (e.g., STS, dPFC) and lower level cortical regions contain information useful for facial expression decoding that go beyond the visual information presented, and implicate a key role for contextual mechanisms such as cortical feedback in facial expression perception under challenging conditions of visual occlusion.

Differences in holistic processing do not explain cultural differences in the recognition of facial expression.

The aim of this study was to investigate the causes of the own-race advantage in facial expression perception. In Experiment 1, we investigated Western Caucasian and Chinese participants' perception and categorization of facial expressions of six basic emotions that included two pairs of confusable expressions (fear and surprise; anger and disgust). People were slightly better at identifying facial expressions posed by own-race members (mainly in anger and disgust). In Experiment 2, we asked whether the own-race advantage was due to differences in the holistic processing of facial expressions. Participants viewed composite faces in which the upper part of one expression was combined with the lower part of a different expression. The upper and lower parts of the composite faces were either aligned or misaligned. Both Chinese and Caucasian participants were better at identifying the facial expressions from the misaligned images, showing interference on recognizing the parts of the expressions created by holistic perception of the aligned composite images. However, this interference from holistic processing was equivalent across expressions of own-race and other-race faces in both groups of participants. Whilst the own-race advantage in recognizing facial expressions does seem to reflect the confusability of certain emotions, it cannot be explained by differences in holistic processing.

Driver’s anger state identification by using facial expression in cooperation with artificial intelligence

Preventive safety system of vehicle is highlighted to reduce the number of traffic accidents. Driver’s state adaptive driving safety system may be one of candidates of the safety system. Identifying driver’s psychosomatic states is indispensable to establish those safety systems. Anger of driver state is often seen in traffic congestion which may be involved in severe traffic accidents. This research adopted Kohonen neural network as classification algorithm to identify anger state of driver by using facial expression. We adopted six types of facial expression which are ordinary, drowsiness, anger, sorrow, delight and surprise according previous research. We classified six types of facial expressions by using KNN. Finally, this research proposes driver’s anger state alert function by using facial expression classification in cooperation with artificial intelligence to prevent potential risks of traffic accidents. Keywords : driver’s anger; facial expression; Kohonen neural network; artificial intelligence.