Facial Expression Database Research Articles

Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods.

Facial expressions of emotion play an important role in human social interactions. However, posed expressions of emotion are not always the same as genuine feelings. Recent research has found that facial expressions are increasingly used as a tool for understanding social interactions instead of personal emotions. Therefore, the credibility assessment of facial expressions, namely, the discrimination of genuine (spontaneous) expressions from posed (deliberate/volitional/deceptive) ones, is a crucial yet challenging task in facial expression understanding. With recent advances in computer vision and machine learning techniques, rapid progress has been made in recent years for automatic detection of genuine and posed facial expressions. This paper presents a general review of the relevant research, including several spontaneous vs. posed (SVP) facial expression databases and various computer vision based detection methods. In addition, a variety of factors that will influence the performance of SVP detection methods are discussed along with open issues and technical challenges in this nascent field.

Emotion Monitoring for Preschool Children Based on Face Recognition and Emotion Recognition Algorithms

In this paper, we study the face recognition and emotion recognition algorithms to monitor the emotions of preschool children. For previous emotion recognition focusing on faces, we propose to obtain more comprehensive information from faces, gestures, and contexts. Using the deep learning approach, we design a more lightweight network structure to reduce the number of parameters and save computational resources. There are not only innovations in applications, but also algorithmic enhancements. And face annotation is performed on the dataset, while a hierarchical sampling method is designed to alleviate the data imbalance phenomenon that exists in the dataset. A new feature descriptor, called “oriented gradient histogram from three orthogonal planes,” is proposed to characterize facial appearance variations. A new efficient geometric feature is also proposed to capture facial contour variations, and the role of audio methods in emotion recognition is explored. Multifeature fusion can be used to optimally combine different features. The experimental results show that the method is very effective compared to other recent methods in dealing with facial expression recognition problems about videos in both laboratory‐controlled environments and outdoor environments. The method performed experiments on expression detection in a facial expression database. The experimental results are compared with data from previous studies and demonstrate the effectiveness of the proposed new method.

MFED: A Database for Masked Facial Expression

Humans sometimes intentionally altered facial expression to mask the genuine emotions for certain purposes. Such masked facial expressions have seldom been recorded. In this study, we constructed a facial expression database in which participants were required to watch emotional video clips and make specific facial expressions which were consistent or inconsistent with the emotional context, and named it Masked Facial Expression Database (MFED). Twenty-four participants were recruited and trained to be familiar with six basic facial expressions. They were instructed to display six expressions under six basic emotional context which resulted in 36 combinations. Seven hundred and eighty three emotional video clips including 36 (6 by 6) categories of facial expressions were collected. We also utilized Facial Action Coding System (FACS) to encode these facial expressions. In addition, local binary patterns from three orthogonal planes (LBP-TOP) and support vector machine (SVM) were employed for feature extraction and subsequent leave-one-subject-out cross validation, respectively. Best performance of classification is 78.80% for masked and non-masked, 26.83% for experienced emotion evoked by video clips, and 47.77% for required (presented by instructions) expressions. For improving the performance of experienced emotion and required expression recognition, a well-designed convolutional neural network (CNN) was also utilized to capture high-level representations of facial expression images. Our clear-cut results demonstrated that masked and non-masked expressions could be moderately discriminated in this database. Experienced emotion was difficult to identify partly because it was definitely covered up by the masked expression.

Photogram Classification-Based Emotion Recognition

This paper presents a method for facial emotion recognition based on parameterized photograms and machine learning techniques. Videos of people displaying emotions are parameterized by a facial feature-based emotional category association process to determine whether a given photogram expresses emotions by comparing the facial action units displayed with findings in the literature about facial emotion. To test the proposed approach, two strategies are adopted. First, photograms displaying emotions are gathered, and then different machine learning classifiers are applied to check the goodness of the obtained set of categorized emotional photograms. Second, classifiers trained on the sets of emotional photograms were then used to emotionally classify all the videos in each database, using all the photograms with no preprocessing or photogram selection. The presented method was tested using the OpenFace parameterizer with emotional videos gathered from Multimedia Understanding Facial Expression (MUG) and Cohn-Kanade (CK+) databases. The outcomes achieved for emotional photogram classification on the sets of emotional photograms reached maximums of 99.80% and 99.63% in the MUG and CK+ databases, respectively. The videos were classified using different voting strategies regarding the outcome of each photogram in the video with all the photogram emotion recognition classifiers obtained results reflecting recognition rates of 70.71% and 66.36% for the videos in MUG and CK+ databases, and reached up to 72.55% and 88.37% when classifier combination strategies were used. The work carried out opens the door to follow-up work concerning data preprocessing and the use of different classifier combination methods in facial emotion recognition.

Facial Expression Recognition with LBP and ORB Features.

Emotion plays an important role in communication. For human–computer interaction, facial expression recognition has become an indispensable part. Recently, deep neural networks (DNNs) are widely used in this field and they overcome the limitations of conventional approaches. However, application of DNNs is very limited due to excessive hardware specifications requirement. Considering low hardware specifications used in real-life conditions, to gain better results without DNNs, in this paper, we propose an algorithm with the combination of the oriented FAST and rotated BRIEF (ORB) features and Local Binary Patterns (LBP) features extracted from facial expression. First of all, every image is passed through face detection algorithm to extract more effective features. Second, in order to increase computational speed, the ORB and LBP features are extracted from the face region; specifically, region division is innovatively employed in the traditional ORB to avoid the concentration of the features. The features are invariant to scale and grayscale as well as rotation changes. Finally, the combined features are classified by Support Vector Machine (SVM). The proposed method is evaluated on several challenging databases such as Cohn-Kanade database (CK+), Japanese Female Facial Expressions database (JAFFE), and MMI database; experimental results of seven emotion state (neutral, joy, sadness, surprise, anger, fear, and disgust) show that the proposed framework is effective and accurate.

Recent trends in facial expression databases including validation processes

Recent trends in facial expression databases including validation processes

Iranian kinect face database (IKFDB): a color-depth based face database collected by kinect v.2 sensor

This study presents a new color-depth based face database gathered from different genders and age ranges from Iranian subjects. Using suitable databases, it is possible to validate and assess available methods in different research fields. This database has application in different fields such as face recognition, age estimation and Facial Expression Recognition and Facial Micro Expressions Recognition. Image databases based on their size and resolution are mostly large. Color images usually consist of three channels namely Red, Green and Blue. But in the last decade, another aspect of image type has emerged, named “depth image”. Depth images are used in calculating range and distance between objects and the sensor. Depending on the depth sensor technology, it is possible to acquire range data differently. Kinect sensor version 2 is capable of acquiring color and depth data simultaneously. Facial expression recognition is an important field in image processing, which has multiple uses from animation to psychology. Currently, there is a few numbers of color-depth (RGB-D) facial micro expressions recognition databases existing. With adding depth data to color data, the accuracy of final recognition will be increased. Due to the shortage of color-depth based facial expression databases and some weakness in available ones, a new and almost perfect RGB-D face database is presented in this paper, covering Middle-Eastern face type. In the validation section, the database will be compared with some famous benchmark face databases. For evaluation, Histogram Oriented Gradients features are extracted, and classification algorithms such as Support Vector Machine, Multi-Layer Neural Network and a deep learning method, called Convolutional Neural Network or are employed. The results are so promising.

Blurred Facial Expression Recognition System by Using Convolution Neural Network

A facial expression is a visual impression of a person's situations, emotions, cognitive activity, personality, intention and psychopathology, it has an active and vital role in the exchange of information and communication between people. In machines and robots which dedicated to communication with humans, the facial expressions recognition play an important and vital role in communication and reading of what is the person implies, especially in the field of health. For that the research in this field leads to development in communication with the robot. This topic has been discussed extensively, and with the progress of deep learning and use Convolution Neural Network CNN in image processing which widely proved efficiency, led to use CNN in the recognition of facial expressions. Automatic system for Facial Expression Recognition FER require to perform detection and location of faces in a cluttered scene, feature extraction, and classification. In this research, the CNN used for perform the process of FER. The target is to label each image of facial into one of the seven facial emotion categories considered in the JAFFE database. JAFFE facial expression database with seven facial expression labels as sad, happy, fear, surprise, anger, disgust, and natural are used in this research. We trained CNN with different depths using gray-scale images from the JAFFE database.The accuracy of proposed system was 100%.

Improving multi-view facial expression recognition through two novel texture-based feature representations

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification.

Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, contrary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNNs) to learn facial features from VGG_Faces, which are then linked to a long short-term memory to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image. As a result, we outperform current state-of-the-art area under the curve performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.

Identity–Expression Dual Branch Network for Facial Expression Recognition

Accurate facial expression recognition is challenging because identity biases introduce large intraclass variations and high interclass similarities. Most existing facial expression recognition approaches are devoted to alleviate the effects of identity. However, based on the theories of cognitive science, psychology, and physiology, this article argues that the identity information is important and can promote expression recognition. Motivated by our investigation of the influences of identity on facial expression recognition, this article proposes an identity–expression dual branch network (IE-DBN) for facial expression recognition. First, identity-related features and expression-related features are learnt from the same input facial expression image by two branches respectively. Then, those two features are aggregated with our bilinear module. The bilinear aggregation not only emphasizes the impact of identity but also improves interclass variations and intraclass similarities. The joint training strategy is proposed to regularize the identity-related features. It can force our network to generate expression-guided identity-related features and suppresses negative identity factors in the same time. Experiments on three popular facial expression databases, including two popular posed facial expression databases and one spontaneous facial expression database, show that our IE-DBN outperforms most of the state of the arts, which demonstrates our superiority in facial expression recognition.

Accurate computing of facial expression recognition using a hybrid feature extraction technique

Facial expression recognition (FER) serves as an essential tool for understanding human emotional behaviors. Facial expressions provide a wealth of information about intentions, emotions, and other inner states. Over the past two decades, the development of an automatic FER device has become one of the most demanding multimedia research areas in human–computer interaction systems. Several automatic systems have been introduced and have achieved precise identification accuracies. Due to the complex nature of the human face, however, problems still exist. Researchers are still struggling to develop effective methods for extracting features from images because of unclear features. This work proposes a methodology that improves high-performance computing in terms of the facial expression recognition accuracy. To achieve the goal of high accuracy, a hybrid method is proposed using the dual-tree m-band wavelet transform (DTMBWT) algorithm based on energy, entropy, and gray-level co-occurrence matrix (GLCM). It is accompanied by the use of a Gaussian mixture model (GMM) as the classification scheme to provide efficient identification of database images in terms of facial expressions. Using the DTMBWT, it is possible to derive many expression features from decomposition levels 1 to 6. Moreover, along with the GLCM features, the contrast and homogeneity features can be retrieved. All the features are eventually categorized and recognized with the aid of the GMM classifier. The proposed algorithms are tested using Japanese Female Facial Expression (JAFFE) database with seven different facial expressions: happiness, sadness, anger, fear, neutral, surprise, and disgust. The results of the experiments show that the highest precision of the proposed technique is 99.53%, which is observed at the 4th decomposition level of the DTMBWT.

Modeling and measuring average nasal asymmetry by dorsum midline and nose tip lateral deviation

In rhinoplasty and nasal reconstruction, achieving symmetry is critical for optimal patient outcomes and reducing re-operation rates. Assessing nasal asymmetry is challenging, both pre- and intra-operatively, if based on only a surgeons' visual perception to assess and adjust the small distances important to cosmesis (<2-3mm). To measure nasal symmetry, we first developed an algorithm to analyze lateral nasal deviation on facial three-dimensional (3D) scans captured by external surface scanning. In this, nasal deviation is measured by first registering a 3D facial scan to orthogonal axes in order to remove tilt. The lateral position of the nasal midline is then found across transverse planes along the dorsum and nasal tip regions by probing midpoints 1 and 2 mm back from the local maximum projection. The nasal deviation measurement algorithm was validated on a simulated asymmetrical nose model with known nasal deviation. Simulated deviations were applied to the symmetrical average nose using an exponential twist away from the face, with control of the maximum deviation and degree of curvature. Modeled deviations were evaluated with the algorithm at clinically negligible (0.02-0.06mm) average differences and for small lateral deviations (1-5mm). Nasal deviation using the algorithms was then measured for the 100 multi-ethnic subjects in the Binghamton University 3D Facial Expression database. Average values for maximum lateral deviation, deviation across the whole nose, and deviation at the nose tip were measured to provide context to deviation measurements in surgical planning. This research presents a new nasal assessment tool that can be useful in improving symmetry in rhinoplasty and reconstruction.

FENP: A Database of Neonatal Facial Expression for Pain Analysis

In this article, we introduce a new neonatal facial expression database for pain analysis. This database, called facial expression of neonatal pain (FENP), contains 11,000 neonatal facial expression images associated with 106 Chinese neonates from two children's hospitals, i.e., the Children's Hospital Affiliated to Nanjing Medical University and Second Affiliated Hospital Affiliated to Nanjing Medical University in China. The facial expression images cover four categories of facial expressions, i.e., severe pain expression, mild pain expression, crying expression and calmness expression, where each category contains 2750 neonatal facial expression images. Based on this database, we also investigate the pain facial expression recognition problem using several state-of-the-art facial expression features and expression recognition methods, such as Gabor+SVM, LBP+SVM, HOG+SVM, LBP+HOG+SVM, and several Convolutional Neural Network (CNN) methods (including AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet). The experimental results indicate that the proposed neonatal pain facial expression database is very suitable for the study of both neonatal pain and facial expression recognition. Moreover, the FENP database is publicly available after signing a license agreement (the users can contact Jingjie Yan (yanjingjie@njupt.edu.cn), Guanming Lu (lugm@njupt.edu.cn)) or Xiaonan Li (xnli@njmu.edu.cn).

A Facial Expression Recognition Model Based on Texture and Shape Features

With the development of computer vision, facial expression recognition has become a research hotspot. To further improve the accuracy of facial expression recognition, this paper probes deep into image segmentation, feature extraction, and facial expression classification. Firstly, the convolution neural network (CNN) was adopted to accurately separate the salient regions from the face image. Next, the Gaussian Markov random field (GMRF) model was improved to enhance the ability of texture features to represent image information, and a novel feature extraction algorithm called specific angle abundance entropy (SAAE) was designed to improve the representation ability of shape features. After that, the texture features were combined with shape features, and trained and classified by the support vector machine (SVM) classifier. Finally, the proposed method was compared with common methods of facial expression recognition on a standard facial expression database. The results show that our method can greatly improve the accuracy of facial expression recognition.

Human and machine validation of 14 databases of dynamic facial expressions

With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli have been developed over the past two decades. The present research aimed to test existing sets of dynamic facial expressions (published between 2000 and 2015) in a cross-corpus validation effort. For this, 14 dynamic databases were selected that featured facial expressions of the basic six emotions (anger, disgust, fear, happiness, sadness, surprise) in posed or spontaneous form. In Study 1, a subset of stimuli from each database (N = 162) were presented to human observers and machine analysis, yielding considerable variance in emotion recognition performance across the databases. Classification accuracy further varied with perceived intensity and naturalness of the displays, with posed expressions being judged more accurately and as intense, but less natural compared to spontaneous ones. Study 2 aimed for a full validation of the 14 databases by subjecting the entire stimulus set (N = 3812) to machine analysis. A FACS-based Action Unit (AU) analysis revealed that facial AU configurations were more prototypical in posed than spontaneous expressions. The prototypicality of an expression in turn predicted emotion classification accuracy, with higher performance observed for more prototypical facial behavior. Furthermore, technical features of each database (i.e., duration, face box size, head rotation, and motion) had a significant impact on recognition accuracy. Together, the findings suggest that existing databases vary in their ability to signal specific emotions, thereby facing a trade-off between realism and ecological validity on the one end, and expression uniformity and comparability on the other.

BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER)

This paper presents a new face descriptor based on binary robust independent elementary features (BRIEF) (Calonder et al. in IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298, 2012; in: European conference on computer vision, Springer, pp 778–792, 2010). The most important properties of BRIEF are the independence of the descriptor length from the matching window size and the possibility of using a subsample of pixel pairs located at arbitrary positions in the matching window. Furthermore, BRIEF is computationally simple and gives a compact representation. The BRIEF descriptor can be used to generate discriminative features globally from an image. However, when BRIEF is used to generate features from a region of an image with no explicit shape such as a face in an image, BRIEF must be used locally to ensure that each pixel in the region is evaluated locally to capture the local properties. In this paper, the performance of BRIEF feature is evaluated in the task of AFER. Using three different facial expression databases, we demonstrate that BRIEF provides satisfactory, encouraging and comparable performance to the performance of alternative face representation tested on the same data, and it is efficient in terms of memory usage and execution time.

Research on an expression classification method based on a probability graph model

There has been continuous development of deep learning models, which has resulted in significant improvement for the level of recognition. Herein we propose an expression classification algorithm that uses a probability graph model to improve the accuracy of learning algorithms on small-scale sample sets. An expression region segmentation method is proposed, which divides the facial expression image into five face regions. According to the theoretical basis for the classification method of the probability map model, an expression classification model based on the model is composed of five expression classification sub-networks and Softmax classification layers. By using these, the expression classification model can be used to classify human facial expression images.The experimental results on the JAFFE facial expression database and the CK expression database demonstrate recognition accuracies of 97.78% and 98.95%, i.e. improvements of 1.85% and 5.92%, respectively. The experimental results show that the proposed method is important for improving the recognition rate of the expressions. In addition, the method effectively improves the analysis and understanding ability for small sample images.

Facial expression recognition based on CNN

Facial expression recognition usually can be defined one of the important research in the field of AI and pattern recognition. Facial expression contains rich invisible information, which can help to understand human emotions and intentions, which has great research value. Aiming at solve problems that usually happened such as low recognition accuracy and weak generalization ability of traditional facial expression recognition methods, this paper introduces the theory of the convolution neural network and establishes the convolution neural network model to realize facial expression recognition. Using the crawler to crawl the network image data and using the Viola-Jones algorithm to detect the original data set, after screening, the face expression database is finally established. The convolution neural network model is applied to face expression recognition of the database. We can find that this method can effectively recognize facial expression, and provides a new way to solve this problem.

Benchmarking commercial emotion detection systems using realistic distortions of facial image datasets

Currently, there are several widely used commercial cloud-based services that attempt to recognize an individual’s emotions based on their facial expressions. Most research into facial emotion recognition has used high-resolution, front-oriented, full-face images. However, when images are collected in naturalistic settings (e.g., using smartphone’s frontal camera), these images are likely to be far from ideal due to camera positioning, lighting conditions, and camera shake. The impact these conditions have on the accuracy of commercial emotion recognition services has not been studied in full detail. To fill this gap, we selected five prominent commercial emotion recognition systems—Amazon Rekognition, Baidu Research, Face++, Microsoft Azure, and Affectiva—and evaluated their performance via two experiments. In Experiment 1, we compared the systems’ accuracy at classifying images drawn from three standardized facial expression databases. In Experiment 2, we first identified several common scenarios (e.g., partially visible face) that can lead to poor-quality pictures during smartphone use, and manipulated the same set of images used in Experiment 1 to simulate these scenarios. We used the manipulated images to again compare the systems’ classification performance, finding that the systems varied in how well they handled manipulated images that simulate realistic image distortion. Based on our findings, we offer recommendations for developers and researchers who would like to use commercial facial emotion recognition technologies in their applications.