Subtle Facial Expressions Research Articles

Mean oriented Riesz features for micro expression classification

Micro-expressions are brief and subtle facial expressions that go on and off the face in a fraction of a second. This kind of facial expressions usually occurs in high stake situations and is considered to reflect a human’s real intent. There has been some interest in micro-expression analysis, however, a great majority of the methods are based on classically established computer vision methods such as local binary patterns, histogram of gradients and optical flow. A novel methodology for micro-expression recognition using the Riesz pyramid, a multi-scale steerable Hilbert transform is presented. In fact, an image sequence is transformed with this tool, then the image phase variations are extracted and filtered as proxies for motion. Furthermore, the dominant orientation constancy from the Riesz transform is exploited to average the micro-expression sequence into an image pair. Based on that, the Mean Oriented Riesz Feature description is introduced. Finally the performance of our methods are tested in two spontaneous micro-expressions databases and compared to state-of-the-art methods.

Solution-Processed Sensing Textiles with Adjustable Sensitivity and Linear Detection Range Enabled by Twisting Structure.

Wearable strain sensors are emerging rapidly for their promising applications in human motion detection for diagnosis, healthcare, training instruction, and rehabilitation exercise assessment. However, it remains a bottleneck in gaining comfortable and breathable devices with the features of high sensitivity, linear response, and tunable detection range. Textiles possess fascinating advantages of good breathability, aesthetic property, tailorability, and excellent mechanical compliance to conformably attach to human body. As the meandering loops in a textile can be extended in different directions, it provides plenty of room for exploring ideal sensors by tuning a twisting structure with rationally selected yarn materials. Herein, textile sensors with twisting architecture are designed via a solution-based process by using a stable water-based conductive ink that is composed of polypyrrole/polyvinyl alcohol nanoparticles with a mean diameter of 50 nm. Depending on the predesigned twisting models, the thus-fabricated textile sensors show adjustable performances, exhibiting a high sensitivity of 38.9 with good linearity and a broad detection range of 200%. Such sensors can be integrated into fabrics and conformably attached to skin for monitoring subtle (facial expressions, breathing, and speaking) and large (stretching, jumping, running and jogging, and sign language) human motions. As a proof-of-concept application, by integrating with a wireless transmitter, the signals detected by our sensors during exercise (e.g., running) can be remotely received and displayed on a smartphone. It is believed that the integration of our textile sensors with selected twisting models into a cloth promises full-range motion detection for wearable electronics and human-machine interfaces.

Recognition of emotion from subtle and non-stereotypical dynamic facial expressions in Huntington's disease

Emotion recognition deficits in Huntington's disease (HD) are well-established. However, most previous studies have measured emotion recognition using stereotypical and intense facial expressions, which are easily recognized and artificial in their appearance. By contrast, everyday expressions are often more challenging to recognize, as they are subtle and non-stereotypical. Therefore, previous studies may have inflated the performance of HD patients and it is difficult to generalize their results to facial expressions encountered in everyday social interactions. In the present study, we tested 21 symptomatic HD patients and 28 healthy controls with a traditional facial expression set, as well as a novel stimulus set which exhibits subtle and non-stereotypical facial expressions. While HD patients demonstrated poor emotion recognition in both sets, when tested with the novel, ecologically looking facial expressions, patients' performance declined to chance level. Intriguingly, patients' emotion recognition deficit was predicted only by the severity of their motor symptoms, not by their cognitive status. This suggests a possible mechanism for emotion recognition impairments in HD, in line with embodiment theories. From this point of view, poor motor control may affect patients' ability to subtly produce and simulate a perceived facial expression, which in turn may contribute to their impaired recognition.

Reading Appetite Cues in Infancy: A Role for Nutrition Education.

Infants communicate complex aspects of appetite through a variety of behaviors. In newborn babies, mouthing and orientation towards odors from breast milk signal preference, whereas in older infants likes and dislikes are signaled via facial expressions. Infants communicate readiness to eat by sucking their hands and eliciting the hunger cry, and indicate satiation through a wide repertoire of behaviors ranging from subtle facial expressions, shifts in gaze through to changes in gesture and bodily movements including turning their head away. Filming mealtime interactions reveals the dynamic nature of infant communication and caregiver response. Optimal responsiveness to infant cues may be influenced by individual characteristics of the mother and the mode of feeding (breastfed or formula). A series of studies has used video capture to characterize the ways in which infants communicate appetite. We have then translated this into an educational resource for healthcare professionals and caregivers on how to identify, interpret, and respond to these cues. There is a potentially important role for nutrition education in promoting both the principles of good nutrition as well as ways to read then respond sensitively to infant appetite cues.

Humans can identify cats’ affective states from subtle facial expressions

AbstractAlthough cats’ popularity as pets rivals that of dogs, cats are little studied, and people's abilities to read this apparently ‘inscrutable’ species have attracted negligible research. To determine whether people can identify feline emotions from cats’ faces, participants (n = 6,329) each viewed 20 video clips of cats in carefully operationalised positively (n =10) or negatively valenced states (n = 10) (cross-factored with low and high activity levels). Obvious cues (eg open mouths or fully retracted ears) were eliminated. Participants’ average scores were low (11.85/20 correct), but overall above chance; furthermore, 13% of participants were individually significantly successful at identifying the valence of cats’ states (scoring ≥ 15/20 correct). Women were more successful at this task than men, and younger participants more successful than older, as were participants with professional feline (eg veterinary) experience. In contrast, personal contact with cats (eg pet-owning) had little effect. Cats in positive states were most likely to be correctly identified, particularly if active rather than inactive. People can thus infer cats’ affective states from subtle aspects of their facial expressions (although most find this challenging); and some individuals are very good at doing so. Understanding where such abilities come from, and precisely how cats’ expressions change with affective state, could potentially help pet owners, animal care staff and veterinarians optimise feline care and welfare.

Ventromedial frontal lobe damage affects interpretation, not exploration, of emotional facial expressions

Recognizing and distinguishing the emotional states of those around us is crucial for adaptive social behavior. Previous work has shown that damage to the ventromedial frontal lobe (VMF) impairs recognition of subtle emotional facial expressions and affects fixation patterns to face stimuli. However, whether this relates to deficits in acquiring or interpreting facial expression information remains unclear. We tested 37 patients with frontal lobe damage, including 17 subjects with VMF lesions, in a series of emotion recognition tasks with different gaze manipulations. Subjects were asked to rate neutral, subtle and extreme emotional expressions while freely examining faces, while instructed to look only at the eyes, and in a gaze-contingent condition that required top-down direction of eye movements to reveal the stimulus. People with VMF damage were worse at detecting subtle disgust during free viewing and confused extreme emotional expressions more than healthy controls. However, fixation patterns did not differ systematically between groups during free or gaze-contingent viewing conditions. Moreover, instruction to fixate only the eyes did not improve the performance of VMF damaged subjects. These data argue that VMF is not necessary for normal fixations to emotional face stimuli, and that impairments in emotion recognition after VMF damage do not stem from impaired information gathering, as indexed by patterns of fixation.

The influence of subtle facial expressions on children’s first impressions of trustworthiness and dominance is not adult-like

Adults’ first impressions of others are influenced by subtle facial expressions; happy faces are perceived as high in trustworthiness, whereas angry faces are rated as low in trustworthiness and high in threat and dominance. Little is known about the influence of emotional expressions on children’s first impressions. Here we examined the influence of subtle expressions of happiness, anger, and fear on children’s implicit judgments of trustworthiness and dominance with the aim of providing novel insights about both the development of first impressions and whether children are able to utilize emotional expressions when making implicit, rather than explicit, judgments of traits. In the context of a computerized storybook, children (4- to 11-year-olds) and adults selected one of two twins (two images of the same identity displaying different emotional expressions) to help them face a challenge; some challenges required a trustworthy partner, and others required a dominant partner. One twin posed a neutral expression, and the other posed a subtle emotional expression of happiness, fear, or anger. Whereas adults were more likely to select a happy partner on trust trials than on dominance trials and were more likely to select an angry partner on dominance trials than on trust trials, we found no evidence that children’s choices reflected a combined influence of desirable trait and emotion. Follow-up experiments involving explicit trait judgments, explicit emotion recognition, and implicit first impression judgments in the context of intense emotional expressions provide valuable insights into the slow development of implicit trait judgments based on first impressions.

Volumetric Changes of the Mid and Lower Face with Animation and the Standardization of Three-Dimensional Facial Imaging.

The objective of this study was to quantify volumetric changes of the mid and lower face caused by facial expression to understand how procedural results measured by three-dimensional imaging can be influenced by lack of standardization. Secondarily, the study identified soft-tissue surface landmarks that can be used to ensure the standardization of three-dimensional images. Three-dimensional facial images of subjects performing 22 facial expressions or changes in head position were captured. Variable degrees of animation during smiling and frowning were also evaluated. Volumetric changes of the malar and jowl regions were quantified using a three-dimensional superimposed image subtraction technique. The translation of 14 standard soft-tissue surface landmarks was assessed during various facial animations to determine which three-dimensional landmarks can be used to standardize three-dimensional images. Twenty subjects participated in the study. Sixteen of the 22 facial expression studies had a significant effect on malar and/or jowl volume. Significant volume changes were noted with subtle animation during smiling and frowning. A combination of five landmarks (i.e., glabella, bilateral cheilion, pogonion, and laryngeal prominence) can be used to standardize three-dimensional images for evaluation of mid and lower facial volume changes. Subtle facial expressions may cause significant volumetric changes in the mid and lower face that can mimic the desired outcomes of surgical and nonsurgical facial rejuvenation procedures. The five-point referencing system allows one to identify subtle changes in head position and facial expression and may aid in the standardization of three-dimensional images.

Necessary Morphological Patches Extraction for Automatic Micro-Expression Recognition

Micro expressions are usually subtle and brief facial expressions that humans use to hide their true emotional states. In recent years, micro-expression recognition has attracted wide attention in the fields of psychology, mass media, and computer vision. The shortest micro expression lasts only 1/25 s. Furthermore, different from macro-expressions, micro-expressions have considerable low intensity and inadequate contraction of the facial muscles. Based on these characteristics, automatic micro-expression detection and recognition are great challenges in the field of computer vision. In this paper, we propose a novel automatic facial expression recognition framework based on necessary morphological patches (NMPs) to better detect and identify micro expressions. Micro expression is a subconscious facial muscle response. It is not controlled by the rational thought of the brain. Therefore, it calls on a few facial muscles and has local properties. NMPs are the facial regions that must be involved when a micro expression occurs. NMPs were screened based on weighting the facial active patches instead of the holistic utilization of the entire facial area. Firstly, we manually define the active facial patches according to the facial landmark coordinates and the facial action coding system (FACS). Secondly, we use a LBP-TOP descriptor to extract features in these patches and the Entropy-Weight method to select NMP. Finally, we obtain the weighted LBP-TOP features of these NMP. We test on two recent publicly available datasets: CASME II and SMIC database that provided sufficient samples. Compared with many recent state-of-the-art approaches, our method achieves more promising recognition results.

The contribution of facial dynamics to subtle expression recognition in typical viewers and developmental visual agnosia

Facial expressions are inherently dynamic cues that develop and change over time, unfolding their affective signal. Although facial dynamics are assumed important for emotion recognition, testing often involves intense and stereotypical expressions and little is known about the role of temporal information in the recognition of subtle, non-stereotypical expressions. In Experiment 1 we demonstrate that facial dynamics are critical for recognizing subtle and non-stereotypical facial expressions, but not for recognizing intense and stereotypical displays of emotion. In Experiment 2 we further examined whether the facilitative effect of motion can lead to improved emotion recognition in LG, an individual with developmental visual agnosia and prosopagnosia, who has poor emotion recognition when tested with static facial expressions. LG's emotion recognition improved when subtle, non-stereotypical faces were dynamic rather than static. However, compared to controls, his relative gain from temporal information was diminished. Furthermore, LG's eye-tracking data demonstrated atypical visual scanning of the dynamic faces, consisting of longer fixations and lower fixation rates for the dynamic-subtle facial expressions, comparing to the dynamic-intense facial expressions. We suggest that deciphering subtle dynamic expressions strongly relies on integrating broad facial regions across time, rather than focusing on local emotional cues, skills which are impaired in developmental visual agnosia.

Morphine reduced perceived anger from neutral and implicit emotional expressions

The μ-opioid system modulates responses to pain and psychosocial stress and mediates non-social and social reward. In humans, the μ-opioid agonist morphine can increase overt attention to the eye-region and visual exploration of faces with neutral expressions. However, little is known about how the human μ-opioid system influences sensitivity to and appraisal of subtle and explicit cues of social threats and reward. Here, we examined the effects of selective μ-opioid stimulation on perception of anger and happiness in faces with explicit, neutral or implicit emotion expressions. Sixty-three healthy adults (32 females) attended two sessions where they received either placebo or 10 mg per oral morphine in randomised order under double-blind conditions. Based on the known μ-opioid reduction of pain and discomfort, as well as reports suggesting that the non-specific partial agonist buprenorphine or the non-specific antagonist naltrexone affect appraisal of social emotional stimuli, we hypothesised that morphine would reduce threat sensitivity and enhance perception of happy facial expressions. While overall perception of others’ happiness was unaffected by morphine treatment, morphine reduced perception of anger in stimuli with neutral and implicit expressions without affecting perception of explicit anger. This effect was statistically unrelated to gender, subjective drug effects, mood and autism trait measures. The finding that a low dose of μ-agonist reduced the propensity to perceive anger in photos with subtle facial expressions is consistent with the notion that μ-opioids mediate social confidence and reduce sensitivity to threat cues.

SMEConvNet: A Convolutional Neural Network for Spotting Spontaneous Facial Micro-Expression From Long Videos

Micro-expression is a subtle and involuntary facial expression that may reveal the hidden emotion of human beings. Spotting micro-expression means to locate the moment when the micro-expression happens, which is a primary step for micro-expression recognition. Previous work in micro-expression spotting focus on spotting micro-expression from short video , and with hand-crafted features. In this paper, we present a methodology for spotting micro-expression from long videos . Specifically, a new convolutional neural network named spotting micro-expression convolutional network was designed for extracting features from video clips, which is the first time that deep learning is used in micro-expression spotting. Then, a feature matrix processing method was proposed for spotting the apex frame from long video , which uses a sliding window and takes the characteristics of micro-expression into account to search the apex frame. Experimental results demonstrate that the proposed method can achieve a better performance than the existing state-of-art methods.

Sensitivity to facial expressions among extremely low birth weight survivors in their 30s.

The current study investigated the impact of birth weight on the ability to recognize facial expressions in adulthood among the longest known prospectively followed cohort of extremely low birth weight survivors (ELBW; <1,000 g). We measured perceptual threshold to detect subtle facial expressions and confusion among different emotion categories in order to disentangle visual perceptual ability from emotional processing. ELBW adults (N = 64, Mage = 31.9 years) were more likely than normal birth weight (NBW) controls (N = 82, Mage = 32.5 years) to see fear in angry faces. This finding was not a result of increased perceptual efficiency in processing fearful expressions in the ELBW adults, since the two groups did not differ on their threshold to detect emotion in low intensity facial expressions. These findings suggest that a processing bias toward fear may reflect long-term developmental effects from being born at ELBW that may portend socioemotional problems that characterize ELBW survivors.

Representational momentum in dynamic facial expressions is modulated by the level of expressed pain: Amplitude and direction effects.

Humans have developed a specific capacity to rapidly perceive and anticipate other people's facial expressions so as to get an immediate impression of their emotional state of mind. We carried out two experiments to examine the perceptual and memory dynamics of facial expressions of pain. In the first experiment, we investigated how people estimate other people's levels of pain based on the perception of various dynamic facial expressions; these differ both in terms of the amount and intensity of activated action units. A second experiment used a representational momentum (RM) paradigm to study the emotional anticipation (memory bias) elicited by the same facial expressions of pain studied in Experiment 1. Our results highlighted the relationship between the level of perceived pain (in Experiment 1) and the direction and magnitude of memory bias (in Experiment 2): When perceived pain increases, the memory bias tends to be reduced (if positive) and ultimately becomes negative. Dynamic facial expressions of pain may reenact an "immediate perceptual history" in the perceiver before leading to an emotional anticipation of the agent's upcoming state. Thus, a subtle facial expression of pain (i.e., a low contraction around the eyes) that leads to a significant positive anticipation can be considered an adaptive process-one through which we can swiftly and involuntarily detect other people's pain.

Brief Report: Is Impaired Classification of Subtle Facial Expressions in Children with Autism Spectrum Disorders Related to Atypical Emotion Category Boundaries?

Impairments in recognizing subtle facial expressions, in individuals with autism spectrum disorder (ASD), may relate to difficulties in constructing prototypes of these expressions. Eighteen children with predominantly intellectual low-functioning ASD (LFA, IQ <80) and two control groups (mental and chronological age matched), were assessed for their ability to classify emotional faces, of high, medium and low intensities, as happy or angry. For anger, the LFA group made more errors for lower intensity expressions than the control groups, classifications did not differ for happiness. This is the first study to find that the LFA group made more across-valence errors than controls. These data are consistent with atypical facial expression processing in ASD being associated with differences in the structure of emotion categories.

Development and application of CatFACS: Are human cat adopters influenced by cat facial expressions?

The domestic cat (Felis silvestris catus) is quickly becoming the most popular animal companion in the world. The evolutionary processes that occur during domestication are known to have wide effects on the morphology, behaviour, cognition and communicative abilities of a species. Since facial expression is central to human communication, it is possible that cat facial expression has been subjected to selection during domestication. Standardised measurement techniques to study cat facial expression are, however, currently lacking. Here, as a first step to enable cat facial expression to be studied in an anatomically based and objective way, CatFACS (Cat Facial Action Coding System) was developed. Fifteen individual facial movements (Action Units), six miscellaneous movements (Action Descriptors) and seven Ear Action Descriptors were identified in the domestic cat. CatFACS was then applied to investigate the impact of cat facial expression on human preferences in an adoption shelter setting. Rehoming speed from cat shelters was used as a proxy for human selective pressure. The behaviour of 106 cats ready for adoption in three different shelters was recorded during a standardised encounter with an experimenter. This experimental setup aimed to mimic the first encounter of a cat with a potential adopter, i.e. an unfamiliar human. Each video was coded for proximity to the experimenter, body movements, tail movements and face movements. Cat facial movements were not related to rehoming speed, suggesting that cat facial expression may not have undergone significant selection. In contrast, rubbing frequency was positively related to rehoming speed. The findings suggest that humans are more influenced by overt prosocial behaviours than subtle facial expression in domestic cats.

Crowdsourcing Empathetic Intelligence

Unobtrusive recognition of the user's mood is an essential capability for affect-adaptive systems. Mood is a subtle, long-term affective state, often misrecognized even by humans. The challenge to train a machine to recognize it from, for example, a video of the user, is significant, and already begins with the lack of ground truth for supervised learning. Existing affective databases consist mainly of short videos, annotated in terms of expressed emotions rather than mood. In very few cases, we encounter perceived mood annotations, of questionable reliability, however, due to the subjectivity of mood estimation and the small number of coders involved. In this work, we introduce a new database for mood recognition from video. Our database contains 180 long, acted videos, depicting typical daily scenarios, and subtle facial and bodily expressions. The videos cover three visual modalities (face, body, Kinect data), and are annotated in terms of emotions (via G-trace) and mood (via the Self-Assessment Manikin and the AffectButton). To annotate the database exhaustively, we exploit crowdsourcing to reach out to an extensive number of nonexpert coders. We validate the reliability of our crowdsourced annotations by (1) adopting a number of criteria to filter out unreliable coders, and (2) comparing the annotations of a subset of our videos with those collected in a controlled lab setting.

Improved categorization of subtle facial expressions modulates Late Positive Potential

Biases in facial expression recognition can be reduced successfully using feedback-based training tasks. Here we investigate with event-related potentials (ERPs) at which stages of stimulus processing emotion-related modulations are influenced by training. Categorization of subtle facial expressions (morphed from neutral to happy, sad or surprise) was trained with correct-response feedback on each trial. ERPs were recorded before and after training while participants categorized facial expressions without response feedback. Behavioral data demonstrated large improvements in categorization of subtle facial expression which transferred to new face models not used during training. ERPs were modulated by training from 450ms post-stimulus onward, characterized by a more gradual increase in P3b/Late Positive Potential (LPP) amplitude as expression intensity increased. This effect was indistinguishable for faces used for training and for new faces. It was proposed that training elicited a more fine-grained analysis of facial information for all subtle expressions, resulting in improved recognition and enhanced emotional motivational salience (reflected in P3b/LPP amplitude) of faces previously categorized as expressing no emotion.

Recognition of subtle and universal facial expressions in a community-based sample of adults classified with intellectual disability.

Across the USA and UK, schemes now exist to aid the successful integration of adults with mild to moderate intellectual disabilities into general society. One factor that may prove important to the success of such schemes is social competence. Here, understanding the facial expressions of others is critical, as emotion recognition is a prerequisite to empathetic responding and an essential factor in social functioning. Yet research in this area is lacking, especially in community-based samples. We investigated the performance of 13 adults with mild to moderate intellectual disability (ID), relative to 13 age-matched controls, on three tasks of emotion recognition (emotion categorisation; recognition of valence; and recognition of arousal), using a number of 'basic' (angry and happy) and more 'subtle' (compassionate and critical) emotional expressions, as well as the posers face in a default relaxed (i.e. 'neutral') state. Importantly, the sample was drawn from a community-based initiative and was therefore representative of populations' government schemes target. Across emotion recognition tasks, the ID adults, as compared with controls, were significantly impaired when labelling the emotions displayed by the poser as well as recognising the associated 'feelings' conveyed by these faces. This was especially true for the neutral, compassionate and angry facial expressions. For example, ID adults demonstrated deficits in categorising neutral and subtle emotional expressions, as well as assessing the valence of such facial expressions. In addition, ID adults also struggled to assess arousal levels; especially those associated with compassionate and angry faces. Given both basic and subtle emotions are conveyed in a range of daily situations, errors in interpreting such facial expressions and, relatedly, understanding what potential behaviours such expressions signify could contribute to the social difficulties ID adults face. This is important because current initiatives such as 'personalisation' do not appear to have schemes supporting training in this area, and understanding the facial expressions of others is, after all, one of our most important non-verbal social communication tools.

Feedback training induces a bias for detecting happiness or fear in facial expressions that generalises to a novel task

Many psychological disorders are characterised by insensitivities or biases in the processing of subtle facial expressions of emotion. Training using expression morph sequences which vary the intensity of expressions may be able to address such deficits. In the current study participants were shown expressions from either happy or fearful intensity morph sequences, and trained to detect the target emotion (e.g., happy in the happy sequence) as being present in low intensity expressions. Training transfer was tested using a six alternative forced choice emotion labelling task with varying intensity expressions, which participants completed before and after training. Training increased false alarms for the target emotion in the transfer task. Hit rate for the target emotion did not increase once adjustment was made for the increase in false alarms. This suggests that training causes a bias for detecting the target emotion which generalises outside of the training task. However it does not increase accuracy for detecting the target emotion. The results are discussed in terms of the training’s utility in addressing different types of emotion processing deficits in psychological disorders.