Automatic Facial Expression Analysis Research Articles

Emotion expressions and cognitive impairments in the elderly: review of the contactless detection approach

The aging population in Canada has been increasing continuously throughout the past decades. Amongst this demographic, around 11% suffer from some form of cognitive decline. While diagnosis through traditional means (i.e., Magnetic Resonance Imagings (MRIs), positron emission tomography (PET) scans, cognitive assessments, etc.) has been successful at detecting this decline, there remains unexplored measures of cognitive health that could reduce stress and cost for the elderly population, including approaches for early detection and preventive methods. Such efforts could additionally contribute to reducing the pressure and stress on the Canadian healthcare system, as well as improve the quality of life of the elderly population. Previous evidence has demonstrated emotional facial expressions being altered in individuals with various cognitive conditions such as dementias, mild cognitive impairment, and geriatric depression. This review highlights the commonalities among these cognitive health conditions, and research behind the contactless assessment methods to monitor the health and cognitive well-being of the elderly population through emotion expression. The contactless detection approach covered by this review includes automated facial expression analysis (AFEA), electroencephalogram (EEG) technologies and heart rate variability (HRV). In conclusion, a discussion of the potentials of the existing technologies and future direction of a novel assessment design through fusion of AFEA, EEG and HRV measures to increase detection of cognitive decline in a contactless and remote manner will be presented.

Human and machine recognition of dynamic and static facial expressions: prototypicality, ambiguity, and complexity.

A growing body of research suggests that movement aids facial expression recognition. However, less is known about the conditions under which the dynamic advantage occurs. The aim of this research was to test emotion recognition in static and dynamic facial expressions, thereby exploring the role of three featural parameters (prototypicality, ambiguity, and complexity) in human and machine analysis. In two studies, facial expression videos and corresponding images depicting the peak of the target and non-target emotion were presented to human observers and the machine classifier (FACET). Results revealed higher recognition rates for dynamic stimuli compared to non-target images. Such benefit disappeared in the context of target-emotion images which were similarly well (or even better) recognised than videos, and more prototypical, less ambiguous, and more complex in appearance than non-target images. While prototypicality and ambiguity exerted more predictive power in machine performance, complexity was more indicative of human emotion recognition. Interestingly, recognition performance by the machine was found to be superior to humans for both target and non-target images. Together, the findings point towards a compensatory role of dynamic information, particularly when static-based stimuli lack relevant features of the target emotion. Implications for research using automatic facial expression analysis (AFEA) are discussed.

This Product Seems Better Now: How Social Media Influencers’ Opinions Impact Consumers’ Post-failure Responses

ABSTRACT Extant research on social media influencers primarily focuses on the prepurchase stage (i.e., how influencers affect consumers’ purchase intentions), while overlooking the postpurchase stage. In this research, we investigate how the valence of an influencer’s product review can impact consumers’ affective states after a dissatisfactory experience with the product. Specifically, two rival hypotheses, based on Social Contagion theory and cognitive dissonance theory, respectively, are tested to assess whether an influencer’s positive (vs. negative) review about the same product can improve consumers’ affect. Results from automatic facial expression analysis and sentiment analysis show that an influencer’s positive (vs. negative) review leads to an improvement in consumers’ post-failure affect. Reconciling the two theories, the authors show that the improvement in consumers’ post-failure affect is a result of their improved attitude toward the product, following an influencer’s positive review. In addition, findings suggest this effect is moderated by the level of trust in the influencer, as well as the severity of the product failure. The results also show an important managerial outcome: An improvement of consumers’ post-failure affect (i.e., happiness) leads to a decrease in consumers’ willingness to share negative word-of-mouth.

Understanding Human Emotions and Detecting Stress Levels using YOLO

Abstract: The emotion recognition system can detect the emotions of a person through their facial expression. Detected emotions can fall into any of the six main data of emotions can fall into any of the six main data of emotions such as happiness, sadness, fear, surprise, disgust, and anger. It is the process of using datasets in recognizing the emotion detected by the system through image processing. Emotion detection leads to maintain a healthy environment in lives. We will use the deep learning technology to generate models with emotion based facial expression to recognized emotions. The datasets contain a large number of labelled facial images or videos with different emotions and poses, enabling researchers to train and test their models effectively. To detect the emotions we have used the YOLOV3 for implementing the existing system and YOLOV7 is used to implement the proposed system. We know that facial expression is one of the most powerful, natural and universal signals for human to convey their emotions states and intentions. Numerous studies have been conducted on automatic facial expression analysis because of its practical important in sociable robotics, medical treatment, driver fatigue surveillance, and many other human-computer interaction systems. In the field of computer vision and machine learning various facial expression recognition (FER) systems have been explored to encode expression information from facial representation. As early as the 21th Ekman and Frisen defined six basic emotions based on cross-culture study, which indicated that humans perceive certain basic emotions in the same way regardless of culture. These prototypical facial expressions are anger, disgust, fear, happiness, sadness, and surprise. Recently advanced research on neuroscience and psychology argued that the model of six basic emotions are culture-specific and not universal.

Enhancing Feature Extraction Technique Through Spatial Deep Learning Model for Facial Emotion Detection

Automatic facial expression analysis is a fascinating and difficult subject that has implications in a wide range of fields, including human–computer interaction and data-driven approaches. Based on face traits, a variety of techniques are employed to identify emotions. This article examines various recent explorations into automatic data-driven approaches and handcrafted approaches for recognising face emotions. These approaches offer computationally complex solutions that provide good accuracy when training and testing are conducted on the same datasets, but they perform less well on the most difficult realistic dataset, FER-2013. The article's goal is to present a robust model with lower computational complexity that can predict emotion classes more accurately than current methods and aid society in finding a realistic, all-encompassing solution for the facial expression system. A crucial step in good facial expression identification is extracting appropriate features from the face images. In this paper, we examine how well-known deep learning techniques perform when it comes to facial expression recognition and propose a convolutional neural network-based enhanced version of a spatial deep learning model for the most relevant feature extraction with less computational complexity. That gives a significant improvement on the most challenging dataset, FER-2013, which has the problems of occlusions, scale, and illumination variations, resulting in the best feature extraction and classification and maximizing the accuracy, i.e., 74.92%. It also maximizes the correct prediction of emotions at 99.47%, and 98.5% for a large number of samples on the CK+ and FERG datasets, respectively. It is capable of focusing on the major features of the face and achieving greater accuracy over previous fashions.

Automatic Facial Expression Analysis as a Measure of User-Designer Empathy

Abstract In human-centered product design and development, understanding the users is essential. Empathizing with the user can help designers gain deeper insights into the user experience and their needs. However, a few studies have captured empathy real time during user interactions. Accordingly, the degree to which empathy occurs and enhances user understanding remains unclear. To narrow this gap, a study was performed exploring the use of video-based facial expression analysis during user interviews, as a means to capture empathy related to understanding vehicle driving experiences under challenging conditions. Mimicry and synchrony have been shown to be predictors of empathy in cognitive psychology. In this study, we adapted this method to study 46 user-designer interviews. The results show that the user and designer exhibited mimicry in their facial expressions, which thereby indicated that affective empathy can be captured via simple video facial recognition. However, we found that the user's facial expressions might not represent their actual emotional tone, which can mislead the designer, and they achieve false empathy. Further, we did not find a link between the observed mimicry of facial expressions and the understanding of mental contents, which indicated that the affective and some cognitive parts of user empathy may not be directly connected. Further studies are needed to understand how facial expression analysis can further be used to study and advance empathic design.

An Exploratory Data Analysis for Synchronous Online Learning Based on AFEA Digital Images

The spread of COVID-19 throughout the world has affected the education sector. In some higher education institution, such as Polytechnic Caltex Riau (PCR), it is mandatory for students to participate in synchronous or asynchronous learning activities via virtual classroom. Synchronous online learning is usually supported by video conferencing media such as Google Meeting or Zoom Meeting. The communication between lecturers and students is captured as an image as evidence of students’ interaction and participation in certain learning subjects. These images can provide information for lecturers in determining students’ internal feelings and measuring students’ interest through facial emotions. Taking this reason into account, the current research aims to analyze the emotions detected in facial expression through images using automatic facial expression analysis (AFEA) and exploratory data analysis (EDA), then visualize the data to determine the possible solution to improve the educational process’ sustainability. The AFEA steps applied were face acquisition to detect facial parts in an image, facial data extraction and representation to process feature extraction on the face, and facial expression recognition to classify faces into emotional expressions. Thus, this paper presents the results obtained from applying machine learning algorithms to classify facial expressions into happy and unhappy emotions with mean values of 5.58 and 2.70, respectively. The data were taken from the second semester of 2020/2021 academic year with 1,206 images. The result highlighted the fact that students showed the facial emotion based on the lecture types, hours, departments, and classes. It indicates that there are, in fact, several factors contributing to the variances of students’ facial emotions classified in synchronous online learning.

Co-Creation with Consumers for Packaging Design Validated through Implicit and Explicit Methods: Exploratory Effect of Visual and Textual Attributes

Packaging is no longer a mere structural element that only aims to preserve foods, but it is also a powerful marketing tool able to affect product perception, purchase decision and consumers’ food choices. Incorporating consumers’ voices into packaging design through co-creation could maximise its impact on the market. The main goal of this exploratory study was to test the usefulness of co-creation with consumers for packaging design. For that purpose, a survey with 200 Spanish participants was conducted to find out which of the presented visual and textual packaging attributes were the most appropriate. A validation study with 40 participants using implicit (eye tracker, galvanic skin response and automatic facial expression analysis) and explicit measurements was used to test the packaging co-created by consumers against some of its possible competitors in the market. The co-creation process with consumers allowed for the identification of the visual and textual attributes, among the available options, that best fit their preferences, whereas the validation process confirmed that the packaging design co-created by consumers was equally or even preferred over the competitors. The information gathered might help designers and marketers to guide the packaging design for fish products in the Spanish market.

Automatic Facial Expression Analysis and Recognition using Zone Based Active and Salient Facial Patches

Recognition of facial expression has many useful applications that have drawn researcher’s interest over the past decade. Extraction of features is a major step in the analysis of expression which leads to fast and accurate recognition of expression. Recognition of facial expressions is not an easy issue for methods of machine learning, as different people can vary in the way they show their expressions and for one expression the image of the same person can differ for brightness, background and position. Recognition of facial expression is therefore still a challenging computer vision problem. In this thesis work, we aim to design a robust technique of automatic facial expression analysis and recognition using zone based active and salient patches of the human face by combining various techniques from computer vision and pattern recognition. Expression recognition is closely related to face recognition where a lot of research has been done and a vast array of algorithms have been introduced. Facial expression recognition (FER) can also be considered as a special case of a pattern recognition problem and many techniques are available. In the designing of an FER system, we divided the system into 4 modules, i.e. preprocessing, active and salient patch extraction and classification. Voila Jones algorithm is used for face detection and after that features are extracted from the facial patches. The active facial patches are located on the facial regions that during different expressions undergo a major change. The active patches are located after detection of facial landmarks and hybrid features are determined from these patches. The use of small parts of face instead of the whole face for extracting features reduces the computational cost. Zoning is applied and got remarkable results. The dimensionality of the function is reduced by using linear discriminant analysis, which is further defined using the support vector machine (SVM). On the basis of classification expression is recognized. We evaluated our algorithm on Extended Cohn-Kanade (CK+) dataset.

SentiNet: A Nonverbal Facial Sentiment Analysis Using Convolutional Neural Network

Human facial expressions are an essential and fundamental component for expressing the state of the human mind. The automatic analysis of these nonverbal facial expressions has become a fascinating and quite challenging problem in computer vision, with its application in different areas, such as psychology, human–machine interaction, health, and augmented reality. Recently, deep learning (DL) has become a widespread technique for studying human nonverbal facial sentiment expressions, and some research attempts have been made to propose a certain model on this topic. The purpose of this paper is to apply the appropriate convolutional neural network (CNN) approach by adding several layers of different dimensions, which allows the CNN approach to efficiently classify human facial sentiment expressions with data augmentation capable of recognizing seven basic human facial expressions: anger, sadness, fear, disgust, happiness, surprise, and neutral. In particular, this study mainly proposes a convolution neural network architecture, as well as learning factors that minimize the memory space and total training time of the proposed network due to the shallow architecture of the model. Following that, we demonstrated our proposed model’s network complexity, computational cost, and classification accuracy on the three benchmark datasets: FER2013, KDEF, and JAFFE. As a result, our proposed approach achieves accuracy of [Formula: see text], [Formula: see text], [Formula: see text] in the FER2013, KDEF, and JAFFE, respectively, which is better compared to other state-of-the-art approaches.

Automatic Coding of Facial Expressions of Pain: Are We There Yet?

Introduction The experience of pain is regularly accompanied by facial expressions. The gold standard for analyzing these facial expressions is the Facial Action Coding System (FACS), which provides so-called action units (AUs) as parametrical indicators of facial muscular activity. Particular combinations of AUs have appeared to be pain-indicative. The manual coding of AUs is, however, too time- and labor-intensive in clinical practice. New developments in automatic facial expression analysis have promised to enable automatic detection of AUs, which might be used for pain detection. Objective Our aim is to compare manual with automatic AU coding of facial expressions of pain. Methods FaceReader7 was used for automatic AU detection. We compared the performance of FaceReader7 using videos of 40 participants (20 younger with a mean age of 25.7 years and 20 older with a mean age of 52.1 years) undergoing experimentally induced heat pain to manually coded AUs as gold standard labeling. Percentages of correctly and falsely classified AUs were calculated, and we computed as indicators of congruency, “sensitivity/recall,” “precision,” and “overall agreement (F1).” Results The automatic coding of AUs only showed poor to moderate outcomes regarding sensitivity/recall, precision, and F1. The congruency was better for younger compared to older faces and was better for pain-indicative AUs compared to other AUs. Conclusion At the moment, automatic analyses of genuine facial expressions of pain may qualify at best as semiautomatic systems, which require further validation by human observers before they can be used to validly assess facial expressions of pain.

Facing Forward: Policy for Automated Facial Expression Analysis

The human face is a powerful tool for nonverbal communication. Technological advances have enabled widespread and low-cost deployment of video capture and facial recognition systems, opening the door for automated facial expression analysis (AFEA). This paper summarizes current challenges to the reliability of AFEA systems and challenges that could arise as a result of reliable AFEA systems. The potential benefits of AFEA are considerable, but developers, prospective users, and policy makers should proceed with caution.

Facial Expression Recognition from Video Sequence Using Self Organizing Feature Map

Automatic analysis of facial expressions is rapidly becoming an area of intense interest in computer vision and artificial intelligence research communities. In this paper an approach is presented for facial expression recognition of the six basic prototype expressions (i.e., joy, surprise, anger, sadness, fear, and disgust) based on Facial Action Coding System (FACS). The approach utilizes the topological ordering patterns produced by Kohonen Self Organizing Map, in which implemented on expression image sequence for each prototype facial expression. The map will compute the topological relationship between the particular expression sequences, starting from the neutral expression to peak. This method tried to find a topological ordering pattern (shape) for each expression; it will not require any pre-processing tedious work such as normalization. The only requirement is that, image background must be kept constant, also with non-rigid head motion. The feature extraction phase had been performed by this method, while the classification phase done by especially designed procedures for shape and direction finding to recognize the pattern of the shape, thereafter the type of the expression also backpropagation neural network is implemented for the classification task. An average recognition rate of 88.7% was achieved for six basic expressions, where different databases had been used for the test of the method.

Efficacy and safety of electroacupuncture treatment in the prevention of negative moods in healthy young men after 30 h of total sleep deprivation: study protocol for a single-center, single-blind, parallel-arm, randomized clinical trial

BackgroundSleep deprivation (SD) among young adults is a major public health concern. In humans, it has adverse effects on mood and results in serious health problems. Faced with SD, persons may take precautionary measures to try and reduce their risk. The aim of this study is to evaluate the efficacy and safety of electroacupuncture (EA) for the prevention of negative moods after SD. In addition, we will do a comparison of the effects of EA on mood after SD at different time points.MethodsThis randomized controlled trial (RCT) will be performed at the First Affiliated Hospital of Changchun University of Chinese Medicine in China. The Standards for Reporting Interventions in Clinical Trials of Acupuncture 2010 will be strictly adhered to. Forty-two healthy male volunteers will be distributed into acupoints electroacupuncture (AE) group, non-acupoints electroacupuncture (NAE) control group, or blank control group. This trial will comprise 1-week baseline (baseline sleep), 1-week preventative treatment, 30-h total sleep deprivation (TSD), and 24-h after waking follow-up period. Participants in the AE group and the NAE control group during the preventative treatment period will be administered with EA treatment once daily for 1 week. Participants in the blank control group will not be administered with any treatment. The primary outcome will be the Profile of Mood States (POMS) Scale. Secondary outcome measures will include changes in the Noldus FaceReader (a tool for automatic analysis of facial expressions) and Positive and Negative Affect Schedule (PANAS) Scale. Total sleep deprivation will be 30 h. During the 30-h TSD period, participants will be subjected to 11 sessions of assessment. Adverse events will be recorded.DiscussionThis study is designed to evaluate the efficacy and safety of EA for the prevention of negative moods after SD. The results of this trial will allow us to compare the effects of EA on mood after SD at different time points. Moreover, the findings from this trial will be published in peer-reviewed journals.Trial registrationChinese Clinical Trial Registry Chi2000039713. Registered on 06 November 2020

Effects of Data Visualization Choices on Psychophysiological Responses, Judgment, and Audit Quality

ABSTRACTWe conduct experiments with Big 4 auditors and business students to investigate psychophysiological responses to Big Data visualizations and the effects of visualization techniques on auditor judgment and audit quality. Results of the first experiment using pupillometry, eye gaze measurements, and automatic facial expression analysis indicate that different visualization techniques produce significant differences in the level of cognitive and emotional arousal. A second experiment investigates whether visualizations that were demonstrated to promote higher and lower levels of arousal have differential effects on auditor judgments and audit quality. In addition, the second experiment investigates whether the reliability of data sources underlying visualizations affect auditors' judgments. Results indicate that visualizations that increase arousal can enhance auditors' ability to recognize disconfirming evidence and incorporate this evidence into their decisions.

Facial expression analysis using local directional stigma mean patterns and convolutional neural networks

This paper represents automatic facial expression analysis method named Local Directional Stigma Mean Patterns (LDSMP) for automatic facial expression analysis and image retrieval using content based facial expression image retrieval and CNN. The traditional local patterns such as Local Binary Patterns (LBP) and Local Ternary Patterns (LTP) are applied for face recognition and expression analysis, calculated using relationship between the center pixel and neighboring pixels. The proposed method calculates the eight directional difference values then divided into the three ranges based on threshold values. Thus, the values are substituted with basic three positive values (+3, +2, +1) and three negative values (-3, -2, -1) to get more sensitive information from an image rather than aforementioned methods. The threshold can be select either static which is selected by user or dynamic is evaluated from image itself and supports to improve the efficiency. The performance of the proposed method is further improved by giving this patterns as input to the Convolutional Neural Networks (CNN) and compared with the existing methods LBP, LTP, and Directional Binary Code (DBC) in terms of Average Precision (AP), Average Recall (AR), and Average Retrieval Rate (ARR) using standard databases COREL 10K (DB1) and JAFFE (The Japanese Female Facial Expression) (DB2) and Extended Cohn-Kanade (CK +) (DB3) dataset.

Action unit classification for facial expression recognition using active learning and SVM

Automatic facial expression analysis remains challenging due to its low recognition accuracy and poor robustness. In this study, we utilized active learning and support vector machine (SVM) algorithms to classify facial action units (AU) for human facial expression recognition. Active learning was used to detect the targeted facial expression AUs, while an SVM was utilized to classify different AUs and ultimately map them to their corresponding facial expressions. Active learning reduces the number of non-support vectors in the training sample set and shortens the labeling and training times without affecting the performance of the classifier, thereby reducing the cost of labeling samples and improving the training speed. Experimental results show that the proposed algorithm can effectively suppress correlated noise and achieve higher recognition rates than principal component analysis and a human observer on seven different facial expressions.

Action Unit Driven Facial Expression Synthesis from a Single Image with Patch Attentive GAN

AbstractRecent advances in generative adversarial networks (GANs) have shown tremendous success for facial expression generation tasks. However, generating vivid and expressive facial expressions at Action Units (AUs) level is still challenging, due to the fact that automatic facial expression analysis for AU intensity itself is an unsolved difficult task. In this paper, we propose a novel synthesis‐by‐analysis approach by leveraging the power of GAN framework and state‐of‐the‐art AU detection model to achieve better results for AU‐driven facial expression generation. Specifically, we design a novel discriminator architecture by modifying the patch‐attentive AU detection network for AU intensity estimation and combine it with a global image encoder for adversarial learning to force the generator to produce more expressive and realistic facial images. We also introduce a balanced sampling approach to alleviate the imbalanced learning problem for AU synthesis. Extensive experimental results on DISFA and DISFA+ show that our approach outperforms the state‐of‐the‐art in terms of photo‐realism and expressiveness of the facial expression quantitatively and qualitatively.

Quantitative Evaluation of Facial Expression in a Patient With Minimally Conscious State After Severe Traumatic Brain Injury.

To investigate whether automatic facial expression analysis can quantify differences in the intensity of facial responses depending on the affective stimuli in a patient with minimally conscious state (MCS). We filmed the facial responses of a patient with MCS during the delivery of three 1-minute auditory stimuli: audio clips of comedy movies, a nurse hilariously talking, and recitation of a novel (comedy, nurse, and recitation conditions, respectively). These measures were repeated at least 13 times for each condition on different days for approximately 10 months. The intensity of being "happy" was estimated from the smiling face using a software called FaceReader. The intensity among 5 conditions including those at 2 resting conditions (pre- and poststimuli) was compared using the Kruskal-Wallis test and the Dunn-Bonferroni test for multiple comparisons. Significantly higher values were found in the intensity of being "happy" in the comedy and nurse conditions versus other conditions, with no significant differences between the recitation and pre- or poststimulus conditions. These findings indicate that the automated facial expression analysis can quantify differences in context-dependent facial responses in the patient recruited in this study. This case study demonstrates the feasibility of using automated facial expression analysis to quantitatively evaluate the differences in facial expressions and their corresponding emotions in a single patient with MCS.

Facial Expression Recognition Using Fast Walidlet Hybrid Transform

Automatic analysis of facial expressions is rapidly becoming an area of intense interest in computer vision and artificial intelligence research communities. In this paper an approach is presented for facial expression recognition of the six basic prototype expressions (i.e., joy, surprise, anger, sadness, fear, and disgust) based on Facial Action Coding System (FACS). The approach is attempting to utilize a combination of different transforms (Walid let hybrid transform); they consist of Fast Fourier Transform; Radon transform and Multiwavelet transform for the feature extraction. Korhonen Self Organizing Feature Map (SOFM) then used for patterns clustering based on the features obtained from the hybrid transform above. The result shows that the method has very good accuracy in facial expression recognition. However, the proposed method has many promising features that make it interesting. The approach provides a new method of feature extraction in which overcome the problem of the illumination, faces that varies from one individual to another quite considerably due to different age, ethnicity, gender and cosmetic also it does not require a precise normalization and lighting equalization. An average clustering accuracy of 94.8% is achieved for six basic expressions, where different databases had been used for the test of the method.