Facial Coding Research Articles

ARL-IL CNN for Automatic Facial Expression Recognition of Infants under 24 Months of Age

Automatic facial expression coding of infants plays an important role in infants-related applications, including computer-aided ASD diagnosis, automatic intervention for ASD children and diagnosis of ADHD, etc. However, most of existing facial expression researches focused on adult facial expression analysis, the infant facial expression recognition has been less investigated. Due to an age gap between the facial expression datasets of adults and infants, a facial expression recognition model trained on adult datasets usually shows poor generalization to infants datasets. A labeled infant facial expression dataset can mitigate this problem, and hence we first collect a facial expression dataset of 30 infants under 24 months of age by recording videos of infants’ facial expression during a face-to-face mother-infant interaction. Due to infants spontaneous facial behaviors, the dataset covers multiple challenges, such as large head-poses, occlusion, facial expression intensities, etc. To develop an automatic facial expression coding system, we propose a framework consisted of adaptive region learning and island loss, i.e., ARL-IL, to self-adaptively discover facial regions with higher discriminability between different emotion classes. The framework was verified on our collected dataset, and attained a classification accuracy of 86.86%, which has shown better performance than conventional method based on hand-crafted features and some basic CNN architectures. To interpret the effectiveness of ARL-IL, we also visualize the learned features and find that the proposed framework focuses on facial regions with more emotion information compared with other hand-crafted features or learned features from basic CNN architectures. The experimental results show that our proposed framework has robustness to the large head-poses and occlusion.

UNDERSTANDING AND IMPROVISATION OF HUMAN PERCEPTION USING ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING

The paper deals with the topic of improvising human perception using Artificial Intelligence to make human beings more efficient and productive. Understanding human perception takes a lot of non-verbal cues such as facial expressions, gesture, body language and tone of voice. Recent research has been made through facial coding and neurofeedback training. To analyse the probable response of a human being at certain expression of emotion, collection of data based on facial expression, vocal utterances, brainwave frequency under challenging condition ssuch as anger, contempt, disgust, fear, sadness and surprise is required. If we can formulate a nalgorithm based on the data collected, then not only it would be possible to calculate certain human action, it can also be possible to change or reduce the chances of success of a certain action. Modern advancements has introduced faster problem solving capability but it has some restrictions, which can be coped by the utilisation of human brain which has far better capabilities.The main concern of this paper is why to use AI and how it will revolutionize the mankind.

Temporal assessment of neonatal pain after airway aspiration.

ObjectiveTo temporally assess a painful stimulus in premature infants using 3 neonatal pain scales.MethodsA total of 83 premature infants were observed during airway aspiration by 3 evaluators (E1, E2 and E3) using 3 pain assessment scales (Neonatal Facial Coding System - NFCS; Neonatal Infant Pain Scale - NIPS; and Premature Infant Pain Profile - PIPP) at 5 time points: T1 (before airway aspiration), T2 (during airway aspiration), T3 (1 minute after airway aspiration), T4 (3 minutes after airway aspiration), and T5 (5 minutes after airway aspiration). Light’s Kappa (agreement among examiners and among scales at each time point) and the McNemar test (comparison among time points) were used considering p < 0.05.ResultsThere was a significant difference between the 3 examiners for T1 and T2 using the 3 scales. In T3, pain was observed in 22.9%/E1, 28.9%/E2, and 24.1%/E3 according to the NFCS; 22.9%/E1, 21.7%/E2, and 16.9%/E3 according to the NIPS; and 49.4%/E1, 53.9%/E2, and 47%/E3 according to the PIPP. There was a difference between T1 and T3 using the 3 scales, except for 2 examiners for the PIPP (E2: p = 0.15/E3: p = 0.17). Comparing T4 and T5 to T1, there was no difference in the 3 scales.ConclusionPremature infants required at least 3 minutes to return to their initial state of rest (no pain).

Evaluation of pain in vaginal and caesarean section birth newborns before and after intramuscular injection

ABSTRACT BACKGROUND AND OBJECTIVES: It is already known that the painful experiences to which the newborn is subjected may be related to future reactions and differences in the behavioral response to pain in newborns from different birth routes. This study aimed to evaluate the response to pain in infants born by cesarean section and vaginal delivery routes. METHODS: This is a prospective cohort study. Newborns born at full term were allocated into two groups: cesarean section and vaginal delivery. The intramuscular vitamin K injection was performed as an acute pain stimulus because it is a routine injection at birth. The newborns were evaluated for pain on two scales, a one-dimensional, the Neonatal Facial Coding System, and a multidimensional, the COMFORT Behavior Scale (Comfort b), in addition to heart rate counting, moments before, immediately after and 10 minutes after the stimuli. Students t-test was used for statistical analyzes, and the level of significance was 5%. RESULTS: Eighty-three children were evaluated. The pain intensity assessed by both scales before the pain stimulus was higher in the cesarean section group, but not statistically significant. Heart rate values after the stimulus showed a statistically significant difference (p<0.05), higher in the cesarean section group than in the vaginal delivery. CONCLUSION: The results did not suggest that there are differences in the perception of pain among newborns born vaginally and cesarean section.

Emotional Responses to Bureaucratic Red Tape

Abstract Although red tape has a long history in public administration research, the emotional consequences of bureaucratic procedures for citizens have received little attention in the literature. Within the framework of behavioral public administration, this article investigates how varying conditions of administrative delay, administrative burden, and rule dysfunctionality in citizen‐state interactions spark discrete emotional reactions. Physiological measurements of emotions (e.g., facial coding, electrodermal activity, heart rate) from 136 participants in a laboratory study show that bureaucratic red tape evokes significant negative emotional responses, especially confusion, frustration, and anger. Experimental evidence also indicates that delay is less stirring than burden, while rule functionality has little placatory effect, regardless of the favorability of outcomes. These results support the conceptualization of red tape as an affective rather than a cognitive phenomenon. They also suggest that negative emotions of citizens are linked to the modus operandi of public administrations.

‘This time with feeling?’ Assessing EU data governance implications of out of home appraisal based emotional AI

The boundaries of personal space and borders of bodily integrity are being tested by deployments of emotional artificial intelligence (EAI) in private and public spaces. By means of sensing, seeing and machine learning of facial expressions, voice, gaze, gestures and range of physiological signals (heart rate, skin conductivity and temperature, muscle activity, body temperature, respiration and other bio-signals), the goal is to make interior emotional life machine-readable for personal, commercial and security objectives.In this paper, we focus on computer vision and face-based analytics to consider the nature, method and development of facial coding, the potential demise of existing approaches, and the rise of even more invasive methods. Criticisms of facial coding have long existed, but recent scholarship and industrial development signals a lack of confidence in ‘basic emotions’ and a turn to appraisal-based accounts of emotion. This inevitably entails use of data about internal physiological and experiential contexts, but also factors external to an individual. To explore this, this paper asks and answers the following question: With regard to deployment in out-of-home situations, what are the legal and privacy implications of appraisal-based emotion capture?

Repeated painful stimuli may affect pain response in neonates of more than 34 weeks post-conceptional age: A case control study

Objective: To evaluate the effect of repeated painful stimuli on pain response in neonates of more than 34 weeks post conceptional age. Method: A case control study was done. Cases comprised neonates more than 34 weeks gestational age having more than ten repeated painful procedures. Controls include age and sex matched neonates with less than three painful procedures. The outcome variables were the duration of cry and Modified Facial Coding Score (MFCS) after intramuscular injection. Results: A total of 100 neonates were enrolled (50 as cases and 50 as controls). Median (interquartile range) of duration of cry in case and control groups was 26 (20-48 seconds) and 30.5 (20-78 seconds) respectively (p >0.05). MFCS score at immediate, 1 and 3 minutes was not statistically significant in the two groups (p >0.05). Conclusions: Exposure to frequent painful stimuli in early neonatal life may not significantly alter pain response in neonates more than 34 weeks post conceptional age. Sri Lanka Journal of Child Health, 2019; 48(3): 251-255

An Automated Research for Emotion Recognition and Generation

As of today, Human Robot Interaction (HRI) is considered as an on-going and challenging trend. The development in this field is vast and still in budding phase with respect to research. Several Algorithms and Models are being designed that allow the robots to precept think and behave like a human being. Scientists are working regressively on creating an ‘Artificial Mind’ as an infrastructure using computational methods which are specifically delineated to guide a robot. Artificial Intelligence has taken the next level which is called as ‘Embodied Artificial Intelligence’ in order to make the humanoids smarter. This research deals with, a practical robot head which recognizes and mirrors the same basic emotions from the user with the presence of visual inputs and Facial Action Code System (FACS) control points on the human face. This research work deals with the design and development of three major modules in which, one module involves the fabrication of the practical robot head to represent the emotions. The next major module is the Artificial Intelligence segment including a Fuzzy Logic. In this segment, the inputs will be taken from the user in the form intensities of visual signals. Based on the Fuzzy Logic classifications; the respective emotion from the user is recognized and fed forward to the practical head. The third module of this research deals with the interfacing mechanism between the Fuzzy Logic module and the Mechanical Humanoid head for traversing the signals generated from the Fuzzy Logic Chunk to the practical robot head in the form mechanical actions. This proposed research model will have a huge impact in the study of interaction between the Human and Robots and research studies on certain critical diseases or syndromes like ASD, Alzheimer’s disease, and Bell’s palsy

Anti-IL-6 Receptor Antibody Inhibits Spontaneous Pain at the Pre-onset of Experimental Autoimmune Encephalomyelitis in Mice.

Chronic pain is a significant symptom in patients with autoimmune encephalomyelitis, such as multiple sclerosis and neuromyelitis optica. The most commonly used animal model of these diseases is experimental autoimmune encephalomyelitis (EAE). We previously reported that evoked pain, such as mechanical allodynia, was improved by an anti-IL-6 receptor antibody in EAE mice. However, few reports have evaluated spontaneous pain in EAE mice. Here, we assessed spontaneous pain in EAE mice by utilizing the Mouse Grimace Scale (MGS, a standardized murine facial expression-based coding system) and evaluated the influence of an anti-IL-6 receptor antibody (MR16-1). EAE was induced in female C57BL/6J mice by subcutaneous immunization with myelin oligodendrocyte glycoprotein 35–55 emulsified in adjuvant and administration of pertussis toxin. Mice were placed individually in cubicles and filmed for about 10 min. Ten clear head shots per mouse from the video recording were given a score of 0, 1, or 2 for each of three facial action units: orbital tightening, nose bulge, and ear position. Clinical symptoms of EAE were also scored. Measurement of 5-HT in the spinal cord and functional imaging of the periaqueductal gray (PAG) were also performed. Compared with control mice, MGS score was significantly higher in EAE mice. MR16-1 prevented this increase, especially in pre-onset EAE mice. Promotion of spinal 5-HT turnover and reduction of PAG activity were observed in pre-onset EAE mice. These results suggest that MR16-1 prevented spontaneous pain developed before EAE onset.

Prenatal opioid exposure heightens sympathetic arousal and facial expressions of pain/distress in term neonates at 24–48 hours post birth

Purpose: The rising issue of opioid use during pregnancy poses an increased risk of fetal exposure to opioids in-utero and the development of neonatal abstinence syndrome (NAS). The cessation of exposure to opioids upon birth causes elevated levels of norepinephrine in the circulation enhancing sympathetic arousal. Skin conductance (SC) detects sympathetic-mediated sweating while the Neonatal Facial Coding System (NFCS) depicts facial expressions of stress and pain. We hypothesize that there will be a direct correlation between SC and NFCS scores, such that neonates with prenatal opioid exposure will have higher SC and facial responses to pain/stress as compared with healthy neonates without prenatal opioid exposure.Objective: This study evaluates the utility of SC and the NFCS in the objective assessment of early postnatal pain response in opioid-exposed and non-opioid exposed neonates.Methods: This prospective, single-center, pilot study enrolled opioid-exposed term neonates (>37 weeks) and healthy controls. Subjects were observed within 24–48 hours post-birth (and prior to opioid withdrawal) for pain at baseline, during, and post-heel lance/squeeze (HLS) with simultaneously measured SC and videotaped facial expressions. SC data included electro-dermal responses over time (EDR/second) and the average amplitude of responses (mean of peaks [MP]). Video data were scored using the NFCS by two trained coders with inter-rater agreement >85%.Results: SC and NFCS scores were significantly associated with both groups. The opioid-exposed neonates had significantly higher skin conductance indices, EDR/sec for the HLS phase, and MP for HLS and post phases as compared with controls (p < .05). Opioid-exposed neonates demonstrated higher NFCS at baseline (p = .003).Conclusions: Prenatal opioid exposure was associated with heightened sympathetic arousal during both pain and recovery phases and higher facial expressions of pain/distress at baseline only. A multimodal system of assessment may be useful in understanding the complexity and severity of opioid withdrawal associated with NAS.

High performance hybrid cognitive framework for bio-facial signal fusion processing for the disease diagnosis

In today’s world, every human’s life is affected by the several numbers of diseases which increases day-by-day due to the unpredicted growth of the pathogens. This leads to the increase in the death rate of the humans in which 70% take place without the proper knowledge of the diagnosis of diseases and care-taking mechanism. Numerous methods have been proposed for the diagnosis of the diseases or predetermination of the diseases. We propose a new method for diagnosing the disease through the fusion of bio-signal and the facial expression codes. The new algorithm which is based on the Cognitive Extreme Learning Machines (CELM) has been implemented for the classification of different facial expressions in accordance with the symptoms of the diseases and relates the results for their diagnosis. Again, the Cognitive Rule Engine has been used for the incorporation for the predetermination and diagnosis. The proposed method has been compared with the existing intelligent learning algorithms and the results are proved to be more accurate in terms of the recognition rate, and training speed.

The effects of alcohol on positive emotion during a comedy routine: A facial coding analysis.

There is considerable interest in understanding the emotional effects of alcohol. While a great deal of experimental research has focused on alcohol's ability to relieve negative emotions, there has been far less focus on the effects of alcohol on positive emotions. Further, the available research on positive emotion tends to test alcohol while participants are alone. Yet alcohol is often consumed in social settings, and enhancing social pleasure is consistently identified as being a primary motive for drinking. We aimed to address this gap in the literature by investigating the impact of alcohol on positive emotional experience in a social setting. We used the Facial Action Coding System (FACS) to examine in a large sample the effects of alcohol on response to comedy in a group setting. Five hundred thirteen social drinkers (51.9% female) were assembled into groups of three unacquainted persons and administered either a moderate dose of alcohol, a placebo, or a nonalcohol control beverage. Following beverage consumption, groups listened to a roughly 5-min comedy clip while their facial expressions were video recorded. More than 5 million frames of video were then FACS-coded. Alcohol consumption enhanced enjoyment (Duchenne) smiles-but not nonenjoyment social smiles-and elevated mood ratings. Results provide multimodal evidence supporting the ability of alcohol to enhance positive emotional experience during a comedy routine delivered in a social context. More broadly, this research illustrates the value of studying emotion in a social context using both self-report and behavior-expressive approaches. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Effectiveness of Hammock Positioning in Reducing Pain and Improving Sleep-Wakefulness State in Preterm Infants.

Hammock positioning is now frequently used with preterm infants admitted to ICUs. However, few studies have investigated the extent to which hammock positioning reduces pain and improves the sleep-wakefulness state compared with traditional positioning. Twenty-six clinically stable newborns with gestational ages from 30 to 37 weeks who were breathing spontaneously were randomly assigned to 2 groups: a hammock-positioning group (n = 13), in which newborns were placed in hammocks in the lateral position, and a traditional-positioning group (n = 13), in which they were kept nested, also in the lateral position. The following variables were evaluated at the beginning and the end of the treatment: pain (with the Premature Infant Pain Profile, and Neonatal Facial Coding System), sleep-wakefulness state (with the Brazelton Neonatal Behavioral Assessment Scale), heart rate, breathing frequency, and peripheral SpO2 . The subjects in the hammock-positioning group showed an improvement in pain compared with the traditional-positioning group(Premature Infant Pain Profile score, 2.62 ± 1.89 vs 2.31 ± 1.97, ΔP = .008) and sleep-wakefulness state score (2.08 ± 0.64 vs 1.23 ± 0.44, Δ P < .001), reduced heart rate (151.69 ± 5.44 vs 142.77 ± 5.18 beats/min, Δ P < .001), breathing frequency (52.31 ± 4.05 vs 50.23 ± 2.55 beats/min, Δ P = .024), and increased peripheral SpO2 (94.69 ± 2.14 vs 98.00 ± 1.22%, Δ P < .001). Hammock positioning was an effective treatment option to reduce pain and improve sleep-wakefulness state. It also helped to reduce heart rate and breathing frequency, and to increase peripheral SpO2 , which made it a treatment option for preterm infants.

Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research.

As early as infancy, caregivers' facial expressions shape children's behaviors, help them regulate their emotions, and encourage or dissuade their interpersonal agency. In childhood and adolescence, proficiencies in producing and decoding facial expressions promote social competence, whereas deficiencies characterize several forms of psychopathology. To date, however, studying facial expressions has been hampered by the labor-intensive, time-consuming nature of human coding. We describe a partial solution: automated facial expression coding (AFEC), which combines computer vision and machine learning to code facial expressions in real time. Although AFEC cannot capture the full complexity of human emotion, it codes positive affect, negative affect, and arousal-core Research Domain Criteria constructs-as accurately as humans, and it characterizes emotion dysregulation with greater specificity than other objective measures such as autonomic responding. We provide an example in which we use AFEC to evaluate emotion dynamics in mother-daughter dyads engaged in conflict. Among other findings, AFEC (a) shows convergent validity with a validated human coding scheme, (b) distinguishes among risk groups, and (c) detects developmental increases in positive dyadic affect correspondence as teen daughters age. Although more research is needed to realize the full potential of AFEC, findings demonstrate its current utility in research on emotion dysregulation.

Using computer-vision and machine learning to automate facial coding of positive and negative affect intensity.

Facial expressions are fundamental to interpersonal communication, including social interaction, and allow people of different ages, cultures, and languages to quickly and reliably convey emotional information. Historically, facial expression research has followed from discrete emotion theories, which posit a limited number of distinct affective states that are represented with specific patterns of facial action. Much less work has focused on dimensional features of emotion, particularly positive and negative affect intensity. This is likely, in part, because achieving inter-rater reliability for facial action and affect intensity ratings is painstaking and labor-intensive. We use computer-vision and machine learning (CVML) to identify patterns of facial actions in 4,648 video recordings of 125 human participants, which show strong correspondences to positive and negative affect intensity ratings obtained from highly trained coders. Our results show that CVML can both (1) determine the importance of different facial actions that human coders use to derive positive and negative affective ratings when combined with interpretable machine learning methods, and (2) efficiently automate positive and negative affect intensity coding on large facial expression databases. Further, we show that CVML can be applied to individual human judges to infer which facial actions they use to generate perceptual emotion ratings from facial expressions.

A Comparative Analysis on Facial Expression in Advertisements -By Utilising Facial Action Coding System(FACS)

A Comparative Analysis on Facial Expression in Advertisements -By Utilising Facial Action Coding System(FACS)

A Comprehensive analysis of Neonatal pain and measures to reduce pain

Pain management of neonates is much debated topic in the medical field. Pain and particularly chronic pain may affect the development of the brain and cause permanent injuries. Neonates being non-verbal are unable to express their pain and thus are exposed to various painful procedures without adequate pain relief. This lack of self-report, inadequate pain assessment tool, lack of awareness and concern amongst the care providers and non-availability of standard protocols lead to inadequate pain relief in neonates. The communication of infant pain is identified through validated tools such as: Premature Infant Pain Profile (PIPP), Neonatal Facial Coding System (NFCS), and the Neonatal Infant Pain Scale (NIPS). The most popular means of reducing minor procedural pain is oral glucose solution, music therapy, cuddling the neonates, breast feeding and use of topical anaesthetic agents. Topical anaesthetic agents that are commonly used like EMLA are not far from side effects. EMLA is known to cause methemoglobinemia in preterm babies. Recently a study has been conducted using 4% Liposomal Lidocaine at a tertiary care centre in India by Kaur et al which has found this agent to be useful to relieve minor procedural pain like Venepuncture and found it to be free from side effects of methaemoglobinaemia.[18] In nutshell, effective management of pain is an ethicals well as a scientific issue. Is it not humane to think about providing adequate pain relief to such tiny non-verbal human beings? Effective pain assessment and management in infants correlates with the quality of healthcare provided and the professionalism shown by healthcare providers. As part of comprehensive pain prevention program, each neonatal unit should develop strategies to provide effective pain relief for all procedures.

AM-FED+: An Extended Dataset of Naturalistic Facial Expressions Collected in Everyday Settings

Public datasets have played a significant role in advancing the state-of-the-art in automated facial coding. Many of these datasets contain posed expressions and/or videos recorded in controlled lab conditions with little variation in lighting or head pose. As such, the data do not reflect the conditions observed in many real-world applications. We present AM-FED+ an extended dataset of naturalistic facial response videos collected in everyday settings. The dataset contains 1,044 videos of which 545 videos (263,705 frames or 21,859 seconds) have been comprehensively manually coded for facial action units. These videos act as a challenging benchmark for automated facial coding systems. All the videos contain gender labels and a large subset (77 percent) contain age and country information. Subject self-reported liking and familiarity with the stimuli are also included. We provide automated facial landmark detection locations for the videos. Finally, baseline action unit classification results are presented for the coded videos. The dataset is available to download online: https://www.affectiva.com/facial-expression-dataset/

Multi-modal pain assessment: are near-infrared spectroscopy, skin conductance, salivary cortisol, physiologic parameters, and Neonatal Facial Coding System interrelated during venepuncture in healthy, term neonates?

BackgroundImproving pain and stress assessments in neonates remains important in preventing the short- and long-term consequences. We aimed to identify the relationships between different pain assessment parameters by simultaneously measuring changes in cortical, autonomic, hormonal, physiological, and behavioral evoked responses to venepuncture in healthy, full-term neonates.MethodsThis observational, prospective study (ancillary to the ACTISUCROSE trial) included 113 healthy, 3-day old, full-term neonates who underwent venepuncture for systematic neonatal screening, from July to October 2013, in a tertiary-level maternity ward of a university hospital. During venepuncture, we simultaneously measured the cortical single-channel near-infrared spectroscopy (NIRS) signals, foot skin conductance, salivary cortisol, physiological responses, and behavioral (Neonatal Facial Coding System [NFCS]) evoked responses.ResultsRegarding the NIRS analysis, the highest correlation was between the NFCS at venepuncture and the change in NIRS integrated values of total hemoglobin (r=0.41, P<0.001) or oxygenated hemoglobin (r=0.27, P<0.001). The NFCS at venepuncture was moderately positively correlated with changes in salivary cortisol (r=0.42, P<0.001) and skin conductance (r=0.29, P<0.001). Salivary cortisol and skin conductance changes were not correlated; the latter parameters were not correlated with heart rate, respiratory rate, or SpO2.ConclusionDuring venepuncture, NFCS was mildly or moderately correlated with salivary cortisol, skin conductance, and cortical NIRS changes.

Surprise: unfolding of facial expressions

ABSTRACTResponses to surprising events are dynamic. We argue that initial responses are primarily driven by the unexpectedness of the surprising event and reflect an interrupted and surprised state in which the outcome does not make sense yet. Later responses, after sense-making, are more likely to incorporate the valence of the outcome itself. To identify initial and later responses to surprising stimuli, we conducted two repetition-change studies and coded the general valence of facial expressions using computerised facial coding and specific facial action using the Facial Action Coding System (FACS). Results partly supported our unfolding logic. The computerised coding showed that initial expressions to positive surprises were less positive than later expressions. Moreover, expressions to positive and negative surprises were initially similar, but after some time differentiated depending on the valence of the event. Importantly, these patterns were particularly pronounced in a subset of facially expressive participants, who also showed facial action in the FACS coding. The FACS data showed that the initial phase was characterised by limited facial action, whereas the later increase in positivity seems to be explained by smiling. Conceptual as well as methodological implications are discussed.