AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Emotional communication is a key element of habilitation care of persons with dementia. It is, therefore, highly preferable for assistive robots that are used to supplement human care provided to persons with dementia, to possess the ability to recognize and respond to emotions expressed by those who are being cared-for. Facial expressions are one of the key modalities through which emotions are conveyed. This work focuses on computer vision-based recognition of facial expressions of emotions conveyed by the elderly. Although there has been much work on automatic facial expression recognition, the algorithms have been experimentally validated primarily on young faces. The facial expressions on older faces has been totally excluded. This is due to the fact that the facial expression databases that were available and that have been used in facial expression recognition research so far do not contain images of facial expressions of people above the age of 65 years. To overcome this problem, we adopt a recently published database, namely, the FACES database, which was developed to address exactly the same problem in the area of human behavioural research. The FACES database contains 2052 images of six different facial expressions, with almost identical and systematic representation of the young, middle-aged and older age-groups. In this work, we evaluate and compare the performance of two of the existing imagebased approaches for facial expression recognition, over a broad spectrum of age ranging from 19 to 80 years. The evaluated systems use Gabor filters and uniform local binary patterns (LBP) for feature extraction, and AdaBoost.MH with multi-threshold stump learner for expression classification. We have experimentally validated the hypotheses that facial expression recognition systems trained only on young faces perform poorly on middle-aged and older faces, and that such systems confuse ageing-related facial features on neutral faces with other expressions of emotions. We also identified that, among the three age-groups, the middle-aged group provides the best generalization performance across the entire age spectrum. The performance of the systems was also compared to the performance of humans in recognizing facial expressions of emotions. Some similarities were observed, such as, difficulty in recognizing the expressions on older faces, and difficulty in recognizing the expression of sadness. The findings of our work establish the need for developing approaches for facial expression recognition that are robust to the effects of ageing on the face. The scientific results of our work can be used as a basis to guide future research in this direction.

Social robots are becoming an integrated part of our daily lives with the goal of understanding humans' social intentions and feelings, a capability which is often referred to as empathy. Despite significant progress towards the development of empathic social agents, current social robots have yet to reach the full emotional and social capabilities. This paper presents our recent effort on incorporating an automated Facial Expression Recognition (FER) system based on deep neural networks into the spoken dialog of a social robot (Ryan) to extend and enrich its capabilities beyond spoken dialog and integrate the user's affect state into the robot's responses. In order to evaluate whether this incorporation can improve social capabilities of Ryan, we conducted a series of Human-Robot-Interaction (HRI) experiments. In these experiments the subjects watched some videos and Ryan engaged them in a conversation driven by user's facial expressions perceived by the robot. We measured the accuracy of the automated FER system on the robot when interacting with different human subjects as well as three social/interactive aspects, namely task engagement, empathy, and likability of the robot. The results of our HRI study indicate that the subjects rated empathy and likability of the affect-aware Ryan significantly higher than non-empathic (the control condition) Ryan. Interestingly, we found that the accuracy of the FER system is not a limiting factor, as subjects rated the affect-aware agent equipped with a low accuracy FER system as empathic and likable as when facial expression was recognized by a human observer.

In this study, we analyzed the facial expressions of the participants from a study where 63 students were learning about Climate Change using Betty’s Brain, half with robotic agents and the other half with on-screen based agents using a Facial Expression Recognition (FER) system. Among FER systems, upon a review of existing offline open source solutions, we chose HyperExtended LightFace (more known as deepface) to extract emotions from the participants’ facial expressions. Based on the extracted emotions from the FER system, we compared the resulting emotions between two groups to investigate if there are differences in displayed emotions due to human robot interaction. The result of this first analysis shows that learners who interacted with robotic agents express fear, happy and neutral emotions more than the students who interacted with on-screen based agents. In addition, neutral emotion was detected more in the robotic condition with a large effect, fear with moderate effect and happy with small effect size. Fear and happiness was 60% correlated, which indicates that the FER system may not distinguish well between these two emotions. The future work entails filtering and validation of face detection results in case of false detection and utilizing different pre-processing methods to detect the face and using different FER algorithms.

Facial expression recognition system is an advanced technology that allows machines to recognize human emotions based on their facial expressions. In order to develop a robust prediction model, this research work proposes three distinct architectural models to produce a facial expression prediction system that looks like this: The first model is on using a support vector machine to carry out a classification task. As a follow-up to the second model, an attempt was made to create a Convolution Neural Network (CNN) using the VGG-NET (Visual Geometry Group Network). Following analysis of the results, an attempt was made to enhance the outcome using the third model, which used convolutional sequential layers linked to seven distinct expressions, and an inference was drawn based on loss and accuracy metric behavior. We will use a dataset of human picture facial images in this research, which has more than 35500 facial photographs and represents seven different types of facial expressions. We will analyze our data and make every effort to remove as much noise as we can before feeding that information to our model. We use the confusion matrix to assess the model’s performance after it has been implemented effectively. To demonstrate the effectiveness of our model architecture, we will generate bar graphs and scatter plots for each model to display model loss and accuracy. The output of this model is visualized with actual class and predictive class and the result has a graphical representation for each and every output facial Images which makes our recognition system user-friendly.

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions.

The facial expression recognition (FER) system is a classifier system that attempts to recognize facial expressions based on the analysis of emotion behaviour on the face. The FER system can be implemented by using one classifier or combining multi feature extraction and/or multi classifiers. In general, FER is used with one classifier system to find the best label. Although a classification system is commonly used to find the most likely facial expression, it still produces substantial numbers of errors due to several factors that influence the FER result, such as data quantity, and environmental conditions (i.e. illumination and noise). Therefore, combined multi feature extraction methods and/or multi classifier systems are useful to avoid the single classifier errors. Multi feature extraction or a multi classifier system combination are used to take advantage of different system hypotheses to find an accurate result. This paper is a survey of the latest system combination techniques being used to enhance the classification performance in the FER system; the most recent studies are presented.

Facial expression recognition (FER) is widely used in applications like neurosciences, affective computing, human-machine interaction, behavioral analysis, etc. However, an application where the use of FER remains completely unexplored is performing arts. It is observed that the classic person independent FER system does not produce satisfactory results on Indian Classical Dance specific dataset due to the different ways of representation of facial expressions in real life and in performing arts. This is our effort to introduce a novel application where classic FER systems (person independent) are used to classify the facial expressions used in Indian Classical Dance. For the person independent system, Local Binary Patterns (LBP) are used as facial feature descriptors and the expressions are classified using Linear SVM. Japanese Female Facial Expressions (JAFFE) is used as a standard dataset to test the FER system and a comparable recognition rate of 83.7% is achieved. The same system is then tested on our dance specific dataset. A rule-based person-dependent system is designed for classification with the help of geometric distances between facial components. The facial component-based approach achieves the recognition rate of 85% on our dataset with 7 classes. This work shows promising results and triggers the exploration of the use of facial expression analysis in performing arts.

The babyhood is a very important stage in the human growth process. Thus learning the facial expression characteristics of infants is of great significance to nursing care for infants. Infants' faces are significantly different from adults' faces, like eyebrows, eyes, nose, cheeks, skin texture, etc. Therefore, the facial expression recognition models trained on adults' facial expression database cannot be directly applied to infants' facial expression recognition. There are dozens of facial expression databases for adults' facial expression analysis, but there is no database composed by various infants' facial expressions. Hence we first collect candid images of infants' facial expression, which includes six obvious expressions of infants, namely crying, curious, happy, sleeping, wronged, neutral. Then the texture and wavelet based PCANet (TW-PCANet) is proposed to fully extract the features of infants' facial expressions. The proposed method first handles original input color images by local binary patterns (LBP) and Gabor wavelet, then the processed images are put into the PCANet to get the final class. The experimental results show that the proposed method can achieve higher recognition rate obtained by PCANet, LBP and Gabor wavelet.

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

Facial expression recognition is a hot research direction of pattern recognition and computer vision. It has been increasingly used in artificial intelligence, human-computer interaction and security monitoring in recent years. Convolution neural network (CNN) as a depth learning architecture can extract the essential features of the image, and in the case of large changes in shooting conditions, its effect is better than the traditional methods of Support Vector Machines(SVM) and Principal Component Analysis(PCA). Therefore, an improved method of facial expression recognition based on CNN is proposed in this paper. The purpose is to classify each facial image as one of the seven facial expressions considered in this study. According to the characteristics of facial expression recognition, a new convolution neural network structure is designed which uses convolution kernel to extract implicit features and max pooling to reduce the dimensions of the extracted implicit features. In comparison to AlexNet network, we can improve the recognition accuracy about 4% higher on the CK+ facial expression database by the aid of Batch Normalization (BN) layer to our network. A facial expression recognition system is constructed in this paper for the convenience of application, and the experimental results show that the system could reach the real-time needs.

Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling

Several research methodologies and human face image databases have been developed based on deliberately produced facial expressions of prototypical emotions. However, real-time and spontaneous facial expression recognition cannot be adequately handled by those existing methods and datasets. To address this problem, research efforts have been made to create spontaneous facial expression image datasets as well as to develop algorithms that can process naturally induced affective behavior. This paper introduces these advances and focuses on a small and specific area of spontaneous facial expression recognition. In this paper, we are concentrating on non-posed image acquisition protocols, which strongly influence the subjects for evoking expressions as natural as possible. We categorize the acquisition protocols into four different parts: image acquisition while playing video games, watching emotional videos, during interviews and from other sources. The taxonomy of facial expression acquisition protocols tells about the typical conditions responsible for producing specific facial expressions in that condition. We also address some important design issues related to spontaneous facial expression recognition systems and list the facial expression databases, which are strictly not acted and non-posed. We also put light on the applications of spontaneously evoked facial expression acquisition and recognition because they have potential medical significance. Moreover, we provide a comprehensive analysis and summary of spontaneous facial expression recognition methods by revealing their pros and cons for future researchers.

Facial expression recognition is very important in many human-robot/human-computer interaction systems. Although so many researches are done, it is hard to find a practical applications in the real world due to its underestimate about individual differences among people. Thus, as a solution for such problem, we introduce a 'personalized' facial expression recognition system. Many previous works on facial expression recognition focus on the well-known six universal facial expressions (happy, sad, fear, angry, surprise and disgust) under usage of unified (or non-separated) classification approach. However, for ordinary people, it is a very difficult task to make such facial expressions without much effort and training. Instead of universal facial expressions, many people show 'personalized' or 'individualized' facial expressions typically. Thus, for dealing with such personalities, we propose a method to construct a personalized classifier based on novel feature selection method. Specifically, feature selection is done by histogram-based approach in the frame of fuzzy neural networks(FNN). Besides, we also use an integrated approach for facial expression recognition. Actual experiments/simulations show that the proposed method is effective not only in view of facial expression recognition but also in view of pattern classifier itself.

PurposeFacial expression provides abundant information for social interaction, and the analysis and utilization of facial expression data are playing a huge driving role in all areas of society. Facial expression data can reflect people's mental state. In health care, the analysis and processing of facial expression data can promote the improvement of people's health. This paper introduces several important public facial expression databases and describes the process of facial expression recognition. The standard facial expression database FER2013 and CK+ were used as the main training samples. At the same time, the facial expression image data of 16 Chinese children were collected as supplementary samples. With the help of VGG19 and Resnet18 algorithm models of deep convolution neural network, this paper studies and develops an information system for the diagnosis of autism by facial expression data.Design/methodology/approachThe facial expression data of the training samples are based on the standard expression database FER2013 and CK+. FER2013 and CK+ databases are a common facial expression data set, which is suitable for the research of facial expression recognition. On the basis of FER2013 and CK+ facial expression database, this paper uses the machine learning model support vector machine (SVM) and deep convolution neural network model CNN, VGG19 and Resnet18 to complete the facial expression recognition.FindingsIn this study, ten normal children and ten autistic patients were recruited to test the accuracy of the information system and the diagnostic effect of autism. After testing, the accuracy rate of facial expression recognition is 81.4 percent. This information system can easily identify autistic children. The feasibility of recognizing autism through facial expression is verified.Research limitations/implicationsThe CK+ facial expression database contains some adult facial expression images. In order to improve the accuracy of facial expression recognition for children, more facial expression data of children will be collected as training samples. Therefore, the recognition rate of the information system will be further improved.Originality/valueThis research uses facial expression data and the latest artificial intelligence technology, which is advanced in technology. The diagnostic accuracy of autism is higher than that of traditional systems, so this study is innovative. Research topics come from the actual needs of doctors, and the contents and methods of research have been discussed with doctors many times. The system can diagnose autism as early as possible, promote the early treatment and rehabilitation of patients, and then reduce the economic and mental burden of patients. Therefore, this information system has good social benefits and application value.