Automatic Facial Expression Recognition Research Articles

Accurate computing of facial expression recognition using a hybrid feature extraction technique

Facial expression recognition (FER) serves as an essential tool for understanding human emotional behaviors. Facial expressions provide a wealth of information about intentions, emotions, and other inner states. Over the past two decades, the development of an automatic FER device has become one of the most demanding multimedia research areas in human–computer interaction systems. Several automatic systems have been introduced and have achieved precise identification accuracies. Due to the complex nature of the human face, however, problems still exist. Researchers are still struggling to develop effective methods for extracting features from images because of unclear features. This work proposes a methodology that improves high-performance computing in terms of the facial expression recognition accuracy. To achieve the goal of high accuracy, a hybrid method is proposed using the dual-tree m-band wavelet transform (DTMBWT) algorithm based on energy, entropy, and gray-level co-occurrence matrix (GLCM). It is accompanied by the use of a Gaussian mixture model (GMM) as the classification scheme to provide efficient identification of database images in terms of facial expressions. Using the DTMBWT, it is possible to derive many expression features from decomposition levels 1 to 6. Moreover, along with the GLCM features, the contrast and homogeneity features can be retrieved. All the features are eventually categorized and recognized with the aid of the GMM classifier. The proposed algorithms are tested using Japanese Female Facial Expression (JAFFE) database with seven different facial expressions: happiness, sadness, anger, fear, neutral, surprise, and disgust. The results of the experiments show that the highest precision of the proposed technique is 99.53%, which is observed at the 4th decomposition level of the DTMBWT.

An Automatic Facial Expression Recognition System Employing Convolutional Neural Network with Multi-strategy Gravitational Search Algorithm

Facial expression recognition (FER) plays a vital role in image processing according to the widespread development of human interactive applications. In the past few years, various researchers have focused on FER for implementing it in different applications. The existing system suffers from various complexities such as low accuracy, computational cost, and poor recognition performances. In this article, we proposed a novel concept to recognize facial expressions. The proposed work comprises three sections that are pre-processing, feature extraction, and classification. The pre-processing techniques remove the unwanted data from the original image and enhance the crucial details for further processing. The Convolutional Neural Network (CNN) is used for feature extraction. But, it yields lower performance in terms of feature extraction due to the shortage of hyperparameter tuning. Hence, the Multi-strategy Gravitational Search Algorithm (M-GSA) is utilized to extract the facial expression features from the eyebrow movement, nose, chin, and lip corner of the facial images. The facial expressions are classified via the Support Vector Machine (SVM) classifier. In this work, the top five facial expressions such as surprise, sad, happy, fear, and angry with three facial expression datasets such as FER-2013 dataset, CK+ dataset, and JAFFE dataset. Ultimately, the proposed method demonstrates better classification accuracy and recognition rates than different kinds of state-of-art methods.

The current challenges of automatic recognition of facial expressions: A systematic review

In recent years, due to its great economic and social potential, the recognition of facial expressions linked to emotions has become one of the most flourishing applications in the field of artificial intelligence, and has been the subject of many developments. However, despite significant progress, this field is still subject to many theoretical debates and technical challenges. It therefore seems important to make a general inventory of the different lines of research and to present a synthesis of recent results in this field. To this end, we have carried out a systematic review of the literature according to the guidelines of the PRISMA method. A search of 13 documentary databases identified a total of 220 references over the period 2014–2019. After a global presentation of the current systems and their performance, we grouped and analyzed the selected articles in the light of the main problems encountered in the field of automated facial expression recognition. The conclusion of this review highlights the strengths, limitations and main directions for future research in this field.

The FaceChannel: A Fast and Furious Deep Neural Network for Facial Expression Recognition

Current state-of-the-art models for automatic facial expression recognition (FER) are based on very deep neural networks that are effective but rather expensive to train. Given the dynamic conditions of FER, this characteristic hinders such models of been used as a general affect recognition. In this paper, we address this problem by formalizing the FaceChannel, a light-weight neural network that has much fewer parameters than common deep neural networks. We introduce an inhibitory layer that helps to shape the learning of facial features in the last layer of the network and, thus, improving performance while reducing the number of trainable parameters. To evaluate our model, we perform a series of experiments on different benchmark datasets and demonstrate how the FaceChannel achieves a comparable, if not better, performance to the current state-of-the-art in FER. Our experiments include cross-dataset analysis, to estimate how our model behaves on different affective recognition conditions. We conclude our paper with an analysis of how FaceChannel learns and adapts the learned facial features towards the different datasets.

Automatic Facial Expression Recognition System Using Shape-Information-Matrix (SIM)

Automatic recognition of facial expressions and modeling of human expressions are very essential in the field of affective computing. The authors have introduced a novel geometric and texture-based method to extract the shapio-geometric features from an image computed by landmarking the geometric locations of facial components using the active appearance model (AAM). Expression-specific analysis of facial landmark points is carried out to select a set of landmark points for each expression to identify features for each specific expression. The shape information matrix (SIM) is constructed the set salient landmark points assign to an expression. Finally, the histogram-oriented gradients (HoG) of SIM are computed which is used for classification with multi-layer perceptron (MLP). The proposed method is tested and validated on four well-known benchmark databases, which are CK+, JAFFE, MMI, and MUG. The proposed system achieved 98.5%, 97.6%, 96.4%, and 97.0% accuracy in CK+, JAFFE, MMI, and MUG database, respectively.

BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER)

This paper presents a new face descriptor based on binary robust independent elementary features (BRIEF) (Calonder et al. in IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298, 2012; in: European conference on computer vision, Springer, pp 778–792, 2010). The most important properties of BRIEF are the independence of the descriptor length from the matching window size and the possibility of using a subsample of pixel pairs located at arbitrary positions in the matching window. Furthermore, BRIEF is computationally simple and gives a compact representation. The BRIEF descriptor can be used to generate discriminative features globally from an image. However, when BRIEF is used to generate features from a region of an image with no explicit shape such as a face in an image, BRIEF must be used locally to ensure that each pixel in the region is evaluated locally to capture the local properties. In this paper, the performance of BRIEF feature is evaluated in the task of AFER. Using three different facial expression databases, we demonstrate that BRIEF provides satisfactory, encouraging and comparable performance to the performance of alternative face representation tested on the same data, and it is efficient in terms of memory usage and execution time.

Attention mechanism-based CNN for facial expression recognition

Facial expression recognition is a hot research topic and can be applied in many computer vision fields, such as human–computer interaction, affective computing and so on. In this paper, we propose a novel end-to-end network with attention mechanism for automatic facial expression recognition. The new network architecture consists of four parts, i.e., the feature extraction module, the attention module, the reconstruction module and the classification module. The LBP features extract image texture information and then catch the small movements of the faces, which can improve the network performance. Attention mechanism can make the neural network pay more attention to useful features. We combine LBP features and attention mechanism to enhance the attention model to obtain better results. In addition, we collected and labelled a new facial expression dataset of seven expressions from 35 subjects aged from 20 to 25. For each subject, we captured both RGB images and depth images with a Microsoft Kinect sensor. For each image type, there are 245 image sequences, each of which contains 110 images, resulting in 26,950 images in total. We apply the newly proposed method to our own dataset and four representative expression datasets, i.e., JAFFE, CK+, FER2013 and Oulu-CASIA. The experimental results demonstrate the feasibility and effectiveness of the proposed method.

Improving the Accuracy of Automatic Facial Expression Recognition in Speaking Subjects with Deep Learning

When automatic facial expression recognition is applied to video sequences of speaking subjects, the recognition accuracy has been noted to be lower than with video sequences of still subjects. This effect known as the speaking effect arises during spontaneous conversations, and along with the affective expressions the speech articulation process influences facial configurations. In this work we question whether, aside from facial features, other cues relating to the articulation process would increase emotion recognition accuracy when added in input to a deep neural network model. We develop two neural networks that classify facial expressions in speaking subjects from the RAVDESS dataset, a spatio-temporal CNN and a GRU cell RNN. They are first trained on facial features only, and afterwards both on facial features and articulation related cues extracted from a model trained for lip reading, while varying the number of consecutive frames provided in input as well. We show that using DNNs the addition of features related to articulation increases classification accuracy up to 12%, the increase being greater with more consecutive frames provided in input to the model.

Automatic facial expression recognition based on MobileNetV2 in Real-time

Facial expression recognition (FER) plays a vital role in human computer interaction and has become important filed of choice for researchers in computer vision and artificial intelligence over the last two decades. As we know, the background or non-face areas of image will seriously affect the accuracy of expression recognition. In the era of mobile networks, the demand for lightweight networks and real-time is growing. However, many expression recognition networks cannot meet the real-time requirements due to excessive parameters and calculations. In order to solve this problems, our proposed methodology combines a supervised transfer learning strategy and a joint supervision method with island loss, which is crucial for facial tasks. In addition, newly proposed Convolutional Neural Network (CNN) model, MobileNetv2, which has both accuracy and speed, is deployed in a real-time framework that enables fast and accurate real-time output. As a result, superior performance to other state-of-the-art methods is achieved in facial expression databases CK+, JAFFE and FER2013.

A robust feature extraction with optimized DBN-SMO for facial expression recognition

Facial expression is the most common technique is used to convey the expressions of human beings. Due to different ethnicity and age, faces differ from one individual to another so that an automatic facial expression analysis and recognition is a difficult operation. To solve this difficulty, this paper proposes a robust feature extraction with optimized DBN-SMO for facial expression recognition (FER). Initially, the pre-processing stage is performed then texture descriptors of Local Phase Quantization (LPQ), Weber Local Descriptor (WLD) and Local Binary Pattern (LBP) are used to extract the features. Moreover, Discrete Cosine Transform (DCT) features are extracted to enhance the recognition rate and reduce the computational cost. After that, the Principal component analysis (PCA) is used for dimension reduction. Finally, a deep belief network (DBN) with Spider monkey optimization (SMO) algorithm is used to classify basic expressions for FER. Here, SMO algorithm is used to optimize bias factors and initial connection weights that control the efficiency of the DBN. The proposed work is performed in the MATLAB environment. Experiments performed on Karolinska Directed Emotional Faces (KDEF), Man-Machine Interaction (MMI), Cohn Kanade (CK+), Extended Denver Intensity of Spontaneous Facial Actions (DISFA+) and Oulu-Chinese Academy of Science Institute of Automation (Oulu-CASIA) datasets and it provides a classification accuracy of 97.93%, 95.42%, 97.58%, 95.76%, and 92.38% respectively, this is higher than other current procedures for seven-class emotion.

A MULTIMODAL APPROACH FOR AUTOMATIC DETECTION OF INFANT PAIN USING FACIAL EXPRESSION AND CRYING

research on automatic detection of facial expressions has been making great strides over the past decade. One challenge in this area has been in not just recognizing the regular facial expressions but in paying attention to the micro expressions or compound expressions i.e. combining multiple expressions which may be critical to the success of an automated system. For instance, detection of pain/no-pain would be of great benefit especially in case of infant pain recognition. This is because the current standard for assessing neonatal pain is discontinuous and inconsistent as it depends highly on the observer’s bias. And sometimes, the drawbacks of manual pain assessment can result in delayed intervention and inconsistent treatment of pain. Hence, there is a need for machine-based techniques to provide consistent and continuous automatic assessment. Preprocessing, pain analysis or feature extraction, and pain recognition are the three main phases in automatic pain detection. Pain recognition from facial gestures is a specific task within the broader task of facial expression recognition i.e. Pain is a compound expression derived from many facial gestures. Hence, pain detection is a very important task from the point of view of computer vision, since it is a clear step towards automatic detection of spontaneous face expressions. The vast majority of the existing methods in the field of automatic facial expressions recognition use Facial Action Coding System (FACS) to detect facial expressions of only adults. However, literature shows that there is ample scope for research in FACS-based method that is designed specifically to detect infants’ facial expression of pain. Furthermore, the challenge in Facial Expression Recognition of Infant pain is because of the fact that the infants’ facial expressions include additional movements and units that are readily not available in the FACS. The purpose of this paper is to conduct a literature survey on the existing research in the area of automated pain analysis and recognition and facilitate in the development of a multi modal pain assessment system for children and infants. Further, this paper discusses how machine learning methods have been used in the Automatic detection of Pain for improving the overall performance. Techniques like Convolutional neural networks (CNNs) and Long-Short Term Memory (LSTM) networks have gained much popularity in the last decades due to the wide range of their applications in medical image analysis and emotion recognition.

ARL-IL CNN for Automatic Facial Expression Recognition of Infants under 24 Months of Age

Automatic facial expression coding of infants plays an important role in infants-related applications, including computer-aided ASD diagnosis, automatic intervention for ASD children and diagnosis of ADHD, etc. However, most of existing facial expression researches focused on adult facial expression analysis, the infant facial expression recognition has been less investigated. Due to an age gap between the facial expression datasets of adults and infants, a facial expression recognition model trained on adult datasets usually shows poor generalization to infants datasets. A labeled infant facial expression dataset can mitigate this problem, and hence we first collect a facial expression dataset of 30 infants under 24 months of age by recording videos of infants’ facial expression during a face-to-face mother-infant interaction. Due to infants spontaneous facial behaviors, the dataset covers multiple challenges, such as large head-poses, occlusion, facial expression intensities, etc. To develop an automatic facial expression coding system, we propose a framework consisted of adaptive region learning and island loss, i.e., ARL-IL, to self-adaptively discover facial regions with higher discriminability between different emotion classes. The framework was verified on our collected dataset, and attained a classification accuracy of 86.86%, which has shown better performance than conventional method based on hand-crafted features and some basic CNN architectures. To interpret the effectiveness of ARL-IL, we also visualize the learned features and find that the proposed framework focuses on facial regions with more emotion information compared with other hand-crafted features or learned features from basic CNN architectures. The experimental results show that our proposed framework has robustness to the large head-poses and occlusion.

Relevance-Based Data Masking: A Model-Agnostic Transfer Learning Approach for Facial Expression Recognition

Deep learning approaches are now a popular choice in the field of automatic emotion recognition (AER) across various modalities. Due to the high costs of manually labeling human emotions however, the amount of available training data is relatively scarce in comparison to other tasks. To facilitate the learning process and reduce the necessary amount of training-data, modern approaches therefore often rely on leveraging knowledge from models that have already been trained on related tasks where data is available abundantly. In this work we introduce a novel approach to transfer learning, which addresses two shortcomings of traditional methods: The (partial) inheritance of the original models structure and the restriction to other neural network models as an input source. To this end we identify the parts in the input that have been relevant for the decision of the model we want to transfer knowledge from, and directly encode those relevant regions in the data on which we train our new model. To validate our approach we performed experiments on well-established datasets for the task of automatic facial expression recognition. The results of those experiments are suggesting that our approach helps to accelerate the learning process.

Determining Subcategories of Facial Expressions for Improved Performance in Practical Applications

In the context of facial expression recognition (FER), this paper reviews the fundamental theories of emotions and further explains the key dimensions of a defined emotional space. The main contribution of this paper is to propose a set of novel categorization methods for facial expressions to be used in the design of an automatic FER system. This novel categorization enables the facial expression to be interpreted in a better way that and to be more effective in practical applications of automatic FER systems. In order to validate the feasibility of the proposed categorization methods, a set of experiments is reported which investigates and analyzes the influence that the novel categorization brings to a multi-view FER system.

Recognition of facial expressions based on CNN features

Facial expressions are a natural way to communicate emotional states and intentions. In recent years, automatic facial expression recognition (FER) has been studied due to its practical importance in many human-behavior analysis tasks such as interviews, autonomous-driving, medical treatment, among others. In this paper we propose a method for facial expression recognition based on features extracted with convolutional neural networks (CNN), taking advantage of a pre-trained model in similar tasks. Unlike other approaches, the proposed FER method learns from mixed instances taken from different databases with the goal of improving generalization, a major issue in machine learning. Experimental results show that the FER method is able to recognize the six universal expressions with an accuracy above 92% considering five of the widely used databases. In addition, we have extended our method to deal with micro-expressions recognition (MER). In this regard, we propose three strategies to create a temporal-aggregated feature vector: mean, standard deviation and early fusion. In this case, the best result is 78.80% accuracy. Furthermore, we present a prototype system that implements the two proposed methods for FER and MER as a tool that allows to analyze videos.

Deep convolution network based emotion analysis towards mental health care

Facial expressions play an important role during communications, allowing information regarding the emotional state of an individual to be conveyed and inferred. Research suggests that automatic facial expression recognition is a promising avenue of enquiry in mental healthcare, as facial expressions can also reflect an individual's mental state. In order to develop user-friendly, low-cost and effective facial expression analysis systems for mental health care, this paper presents a novel deep convolution network based emotion analysis framework to support mental state detection and diagnosis. The proposed system is able to process facial images and interpret the temporal evolution of emotions through a new solution in which deep features are extracted from the Fully Connected Layer 6 of the AlexNet, with a standard Linear Discriminant Analysis Classifier exploited to obtain the final classification outcome. It is tested against 5 benchmarking databases, including JAFFE, KDEF,CK+, and databases with the images obtained ‘in the wild’ such as FER2013 and AffectNet. Compared with the other state-of-the-art methods, we observe that our method has overall higher accuracy of facial expression recognition. Additionally, when compared to the state-of-the-art deep learning algorithms such as Vgg16, GoogleNet, ResNet and AlexNet, the proposed method demonstrated better efficiency and has less device requirements. The experiments presented in this paper demonstrate that the proposed method outperforms the other methods in terms of accuracy and efficiency which suggests it could act as a smart, low-cost, user-friendly cognitive aid to detect, monitor, and diagnose the mental health of a patient through automatic facial expression analysis.

Fast and Efficient Facial Expression Recognition Using a Gabor Convolutional Network

Automatic facial expression recognition (FER) is a fundamental topic in computer vision. Many studies have indicated that facial emotion changes are strongly related to certain regions of interest (ROIs), such as the mouth, eyes, eyebrows, and nose; therefore, the features of these facial ROIs are very important for identifying expressions. Since Gabor filters are very efficient in extracting visual content, Gabor orientation filters (GoFs) modulated by Gabor kernels and traditional convolutional filters can capture such ROI information better than conventional convolutional filters. Consequently, this letter presents a light Gabor convolutional network (GCN) consisting of only four Gabor convolutional layers and two linear layers for FER tasks. Extensive experiments on the FER2013, FERPlus and Real-world Affective Faces (RAF) databases demonstrate that the proposed method achieves good recognition accuracy and requires very low computational costs. The source code can be found at https://github.com/general515/Facial_Expression_Recognition_Using _GCN.

Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition.

Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem in the following aspects. First, to stimulate the research of FER under real-world occlusions and variant poses, we annotate several in-the-wild FER datasets with pose and occlusion attributes for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

Facial Expression Recognition (FER) is a challenging task that improves natural human-computer interaction. This paper focuses on automatic FER on a single in-the-wild (ITW) image. ITW images suffer real problems of pose, direction, and input resolution. In this study, we propose a pyramid with super-resolution (PSR) network architecture to solve the ITW FER task. We also introduce a prior distribution label smoothing (PDLS) loss function that applies the additional prior knowledge of the confusion about each expression in the FER task. Experiments on the three most popular ITW FER datasets showed that our approach outperforms all the state-of-the-art methods.

Multi-pose facial expression recognition using appearance-based facial features

The problem of automatic facial expression recognition is both interesting and challenging with a strong impact on many application areas such as animation and human-computer interaction. The field has shown tremendous growth over the past years with its benchmarking efforts and progress. In this paper, an automated system to recognise facial expressions using the deep convolutional neural network is presented. The proposed system has used the appearance based features to recognise the facial expressions from imagery data with pose variations. The appearance based features are extracted by implementing an integrated approach of Gabor filter with local binary pattern method and the selection process of extracted features is executed using the concept of principal component analysis (PCA). The proposed system performance is accessed using the evaluation metrics of precision, recall, f-measure, and recognition rate for the frontal and half side pose images of the JAFFE and KDEF datasets.