Expression Recognition Method Research Articles

Research on facial expression recognition of video stream based on OpenCV

In order to overcome the poor performance of expression similarity measurement in traditional video stream facial expression recognition methods, an OpenCV based facial expression recognition method is proposed. In this method, the video stream face detection image is obtained by the window detection of various features in each position for the video stream image through the cascade classifier, and the image preprocessing is implemented. Based on OpenCV, the most important eyes and mouth in the facial expression are modeled, the eye feature model and mouth feature model are constructed, and the facial expression recognition of the video stream is realised through the constructed model. The experimental results show that the performance of expression similarity measurement is better, and the recognition rate of different expressions is more than 90%.

Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom

The automatical recognition of human facial expression has attracted attention in the field of computer vision and machine learning. Previous works on this topic set many constraints, such as the impact caused by restricted scenarios and low image quality. To address those problems, we propose a new infrared facial expression recognition method with multi-label distribution learning for understanding non-verbal behaviors in the classroom. Specifically, we first compute the feature similarities of seven basic facial expressions to describe the relationship among the adjacent expression images. Then, the similarity values are fitted by a Cauchy distribution function. Furthermore, we construct a new deep network with Cauchy distribution-based label learning (CDLLNet), instead of the conventional single expression labels. By these revised labels, one infrared facial expression can contribute to the learning of neighboring expression labels, as well as its real expression label. The performance of proposed network is evaluated on two facial expression datasets: Oulu-CASIA and CK+. Several qualitative and quantitative experimental results verify that the CDLLNet network can achieve robust results and significantly outperforms the existing state-of-the-art facial expression algorithms.

Dynamic Objectives Learning for Facial Expression Recognition

Facial expression recognition has been widely used to solve the problems such as lie detection and human-machine interaction. However, due to the difficulties to control the application environments, current methods have the lower recognition accuracy in practice. This paper proposes a new method for facial expression recognition by considering several aspects. First, human beings are easy to recognize some expressions, while difficult to recognize others. Inspired by this intuition, a new loss function is proposed to enlarge the distances between samples from easily confused categories. Second, human learning is divided into many stages, and the learning objective of each stage is different. Thus, dynamic objectives learning is proposed, where each objective at different stage is defined by the corresponding loss function. In order to better realize the above ideas, a new deep neural network for facial expression recognition is proposed, which integrates the covariance pooling layer and residual network units into the deep convolution neural network so as to better perform dynamic objectives learning. The experimental results on the standard databases verify the effectiveness and the superior performance of our methods.

A Facial Expression Recognition Method Using Improved Capsule Network Model

Aiming at the problem of facial expression recognition under unconstrained conditions, a facial expression recognition method based on an improved capsule network model is proposed. Firstly, the expression image is normalized by illumination based on the improved Weber face, and the key points of the face are detected by the Gaussian process regression tree. Then, the 3dmms model is introduced. The 3D face shape, which is consistent with the face in the image, is provided by iterative estimation so as to further improve the image quality of face pose standardization. In this paper, we consider that the convolution features used in facial expression recognition need to be trained from the beginning and add as many different samples as possible in the training process. Finally, this paper attempts to combine the traditional deep learning technology with capsule configuration, adds an attention layer after the primary capsule layer in the capsule network, and proposes an improved capsule structure model suitable for expression recognition. The experimental results on JAFFE and BU-3DFE datasets show that the recognition rate can reach 96.66% and 80.64%, respectively.

FENP: A Database of Neonatal Facial Expression for Pain Analysis

In this article, we introduce a new neonatal facial expression database for pain analysis. This database, called facial expression of neonatal pain (FENP), contains 11,000 neonatal facial expression images associated with 106 Chinese neonates from two children's hospitals, i.e., the Children's Hospital Affiliated to Nanjing Medical University and Second Affiliated Hospital Affiliated to Nanjing Medical University in China. The facial expression images cover four categories of facial expressions, i.e., severe pain expression, mild pain expression, crying expression and calmness expression, where each category contains 2750 neonatal facial expression images. Based on this database, we also investigate the pain facial expression recognition problem using several state-of-the-art facial expression features and expression recognition methods, such as Gabor+SVM, LBP+SVM, HOG+SVM, LBP+HOG+SVM, and several Convolutional Neural Network (CNN) methods (including AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet). The experimental results indicate that the proposed neonatal pain facial expression database is very suitable for the study of both neonatal pain and facial expression recognition. Moreover, the FENP database is publicly available after signing a license agreement (the users can contact Jingjie Yan (yanjingjie@njupt.edu.cn), Guanming Lu (lugm@njupt.edu.cn)) or Xiaonan Li (xnli@njmu.edu.cn).

A Facial Expression Recognition Model Based on Texture and Shape Features

With the development of computer vision, facial expression recognition has become a research hotspot. To further improve the accuracy of facial expression recognition, this paper probes deep into image segmentation, feature extraction, and facial expression classification. Firstly, the convolution neural network (CNN) was adopted to accurately separate the salient regions from the face image. Next, the Gaussian Markov random field (GMRF) model was improved to enhance the ability of texture features to represent image information, and a novel feature extraction algorithm called specific angle abundance entropy (SAAE) was designed to improve the representation ability of shape features. After that, the texture features were combined with shape features, and trained and classified by the support vector machine (SVM) classifier. Finally, the proposed method was compared with common methods of facial expression recognition on a standard facial expression database. The results show that our method can greatly improve the accuracy of facial expression recognition.

Deep Learning Based Approach of Emotion Detection and Grading System

In the past two decades, there has been an increasing research in recognition and analysis of human emotions in areas of neuroscience, psychology, cognitive science, and computer science. Facial expressions play an important role in assessment of several neuropsychiatric disorders where patients are having impairment in recognition as well as expression of emotional facial expressivity. The majority of the previous work in the domain of machine analysis of human emotion was focused on recognition of prototypic expressions of six basic emotions or also on finding out movements of the individual muscle that the face can produce. Most of these automated facial expression recognition methods were based on data that has been posed on demand and acquired in laboratory settings. But, for real-world applications, instead of classifying a face image into one of the facial expression categories, focus needs to be given on the problem of estimating the depth of facial expression for further grading the emotions leading to applicability in various clinical research. In this paper we have proposed a deep learning-based Emotion Detection and Grading System (D-EDGS) for estimation of facial emotional expression into one of the seven basic categories of emotion. Along with this the proposed model D-EDGS also works towards grading the emotions into low, medium and high scale. In future, effectiveness of D-EDGS can be applied for the development of decision support system in assessment of neuro psychiatric disorders.

Facial expression recognition method based on a joint normalization strategy

Facial expression recognition method based on a joint normalization strategy

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Facial expression recognition from video sequences is currently an interesting research topic in computer vision, pattern recognition, artificial intelligence, etc. Considering the problem of semantic gap between the extracted hand-designed features in affective videos and subjective emotions, recognizing facial expressions from video sequences is a challenging subject. To tackle this problem, this paper proposes a new method of facial expression recognition from video sequences via deep residual attention network. Firstly, due to the difference in the intensity of emotional representation of each local area in a facial image, deep residual attention networks are employed to learn high-level affective expression features for each frame of facial expression images in video sequences. The used deep residual attention networks integrate deep residual networks with a spatial attention mechanism. Then, average-pooling is performed to produce fixed-length global video-level feature representations. Finally, the global video-level feature representations are utilized as inputs of a multi-layer perceptron to conduct facial expression classification tasks in video sequences. Experimental results on two public video emotional datasets, i.e. BAUM-1s and RML, demonstrate the effectiveness of the proposed method.

A Facial Expression Recongnition Method Based on Dlib, RI-LBP and ResNet

A method of facial expression recognition using a composite feature is proposed. The method combines the expanded Dlib facial feature detector, the rotation-invariant local binary pattern (RI-LBP) and the 50-layer ResNet neural network model (ResNet_50). First, the expanded Dlib was used to locate 83 feature points on the face, obtainting the Dlib feature after preprocessing and dimentionality reduction (PCA). Then, the rotation-invariant LBP feature was extracted from 8 important regions after tilt correction. Furthermore, a 50-layer ResNet neural network was used to extract the low level features from the images. Finally, the three features were combined and extreme learning machine (ELM) was used to classify the composite facial features. The experimental results on Jaffe and CK+ datasets showed that the proposed method performs better compared with other methods.

Facial expression recognition based on CNN

Facial expression recognition usually can be defined one of the important research in the field of AI and pattern recognition. Facial expression contains rich invisible information, which can help to understand human emotions and intentions, which has great research value. Aiming at solve problems that usually happened such as low recognition accuracy and weak generalization ability of traditional facial expression recognition methods, this paper introduces the theory of the convolution neural network and establishes the convolution neural network model to realize facial expression recognition. Using the crawler to crawl the network image data and using the Viola-Jones algorithm to detect the original data set, after screening, the face expression database is finally established. The convolution neural network model is applied to face expression recognition of the database. We can find that this method can effectively recognize facial expression, and provides a new way to solve this problem.

Human Emotion Recognition Based on Face and Facial Expression Detection Using Deep Belief Network Under Complicated Backgrounds

The human emotion recognition based on facial expression has a significant meaning in the application of intelligent man–machine interaction. However, the human face images vary largely in real environments due to the complex backgrounds and luminance. To solve this problem, this paper proposes a robust face detection method based on skin color enhancement model and a facial expression recognition algorithm with block principal component analysis (PCA). First, the luminance range of human face image is broadened and the contrast ratio of skin color is strengthened by the homomorphic filter. Second, the skin color enhancement model is established using YCbCr color space components to locate the face area. Third, the feature based on differential horizontal integral projection is extracted from the face. Finally, the block PCA with deep neural network is used to accomplish the facial expression recognition. The experimental results indicate that in the case of weaker illumination and more complicated backgrounds, both the face detection and facial expression recognition can be achieved effectively by the proposed algorithm, meanwhile the mean recognition rate obtained by the facial expression recognition method is improved by 2.7% comparing with the traditional Local Binary Patterns (LBPs) method.

Discriminative deep multi-task learning for facial expression recognition

Deep multi-task learning (DMTL) is an efficient machine learning technique that has been widely utilized for facial expression recognition. However, current deep multi-task learning methods typically only consider the information of class labels, while ignoring the local information of sample spatial distribution. In this paper, we propose a discriminative DMTL (DDMTL) facial expression recognition method, which overcomes the above shortcomings by considering both the class label information and the samples’ local spatial distribution information simultaneously. We further design a siamese network to evaluate the local spatial distribution through an adaptive reweighting module, utilizing the class label information with different confidences. In addition, by taking the advantage of the provided local distribution information of samples, DDMTL is able to achieve acceptable results even if the number of training samples is small. We implement experiments on three facial expression datasets. The experimental results demonstrate that DDMTL is superior to the state-of-the-art methods.

Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features

In spite of the recent advancements in the field of deep learning based techniques for facial expression recognition, the efficiency of the state-of-the-art recognition methods in the wild scenarios, remains a challenge. The main reason behind the less efforts made for handling wild scenarios is two-folds: very less and varying levels of cues available to identify the distinguishable patterns of features (spatial and temporal) and non-availability of a big dataset to train a deep learning model. Recently, a huge dataset called AffectNet is introduced in the literature providing enough base to apply a deep learning model to train. This paper proposes an efficient combination of hand crafted and deep learning features for facial expression recognition in the wild. We use facial landmark points as hand-crafted features and XceptionNet for the deep learned features. We experiment with XceptionNet and Densenet propose the use of XceptionNet as it performs better compared to DenseNet, when applied on wild scenarios. The proposed fusion of the hand-crafted and XceptionNet features outperforms the state-of-the-art methods for facial expression recognition in the wild.

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

The convolutional neural network (CNN) has been widely used in image recognition field due to its good performance. This paper proposes a facial expression recognition method based on the CNN model. Regarding the complexity of the hierarchic structure of the CNN model, the activation function is its core, because the nonlinear ability of the activation function really makes the deep neural network have authentic artificial intelligence. Among common activation functions, the ReLu function is one of the best of them, but it also has some shortcomings. Since the derivative of the ReLu function is always zero when the input value is negative, it is likely to appear as the phenomenon of neuronal necrosis. In order to solve the above problem, the influence of the activation function in the CNN model is studied in this paper. According to the design principle of the activation function in CNN model, a new piecewise activation function is proposed. Five common activation functions (i.e., sigmoid, tanh, ReLu, leaky ReLus and softplus–ReLu, plus the new activation function) have been analysed and compared in facial expression recognition tasks based on the Keras framework. The Experimental results on two public facial expression databases (i.e., JAFFE and FER2013) show that the convolutional neural network based on the improved activation function has a better performance than most-of-the-art activation functions.

An exemplar pyramid feature extraction based humerus fracture classification method

Humerus fracture have been widely seen disease in the orthopedic clinics and classification of them is a hard process for orthopedist. The main aim of the proposed method is to classify humerus fracture by using a naïve and multileveled method. We collected a novel humerus fracture X-ray image dataset. This dataset consists of 115 images. In this paper, a novel stable feature extraction method is presented to classify humerus fractures. This method is called exemplar pyramid method and it is inspired by exemplar facial expression recognition methods. To classify humerus fractures, X-ray images were employed as input. In this study, X-ray images are resized to 512 × 512 sized image. Then, the used humerus fracture images are divided into 64 × 64 size of exemplars. To create levels, maximum pooling which has been mostly used in deep networks is used and four levels are created. Histogram of oriented gradients (HOG) and local binary pattern (LBP) are employed for feature generation. The most discriminative ones of the generated and concatenated features are selected by using ReliefF and Neighborhood Component Analysis (NCA) based two levelled feature selector (RFNCA).To emphasize success of the proposed exemplar pyramid model based feature generation, four conventional classifiers are chosen for classification and the proposed exemplar pyramid model achieved 99.12% classification accuracy by using leave one out cross validation (LOOCV). Results and tests clearly illustrates success of the proposed exemplar pyramid model based humerus fracture classification method. The results also shown that the proposed exemplar pyramid model achieved higher classification rate than Orthopedist specialized in shoulder.

Facial Expression Recognition by Regional Weighting with Approximated Q-Learning

Several facial expression recognition methods cluster facial elements according to similarity and weight them considering the importance of each element in classification. However, these methods are limited by the pre-definitions of units restricting modification of the structure during optimization. This study proposes a modified support vector machine classifier called Grid Map, which is combined with reinforcement learning to improve the classification accuracy. To optimize training, the input image size is normalized according to the cascade rules of a pre-processing detector, and the regional weights are assigned by an adaptive cell size that divides each region of the image using bounding grids. Reducing the size of the bounding grid reduces the area used for feature extraction, allowing more detailed weighted features to be extracted. Error-correcting output codes with a histogram of gradient is selected as the classification method via an experiment to determine the optimal feature and classifier selection. The proposed method is formulated into a decision process and solved via Q-learning. To classify seven emotions, the proposed method exhibits accuracies of 96.36% and 98.47% for four databases and Extended Cohn-–Kanade Dataset (CK+), respectively. Compared to the basic method exhibiting a similar accuracy, the proposed method requires 68.81% fewer features and only 66.33% of the processing time.

Recognition of facial expressions based on CNN features

Facial expressions are a natural way to communicate emotional states and intentions. In recent years, automatic facial expression recognition (FER) has been studied due to its practical importance in many human-behavior analysis tasks such as interviews, autonomous-driving, medical treatment, among others. In this paper we propose a method for facial expression recognition based on features extracted with convolutional neural networks (CNN), taking advantage of a pre-trained model in similar tasks. Unlike other approaches, the proposed FER method learns from mixed instances taken from different databases with the goal of improving generalization, a major issue in machine learning. Experimental results show that the FER method is able to recognize the six universal expressions with an accuracy above 92% considering five of the widely used databases. In addition, we have extended our method to deal with micro-expressions recognition (MER). In this regard, we propose three strategies to create a temporal-aggregated feature vector: mean, standard deviation and early fusion. In this case, the best result is 78.80% accuracy. Furthermore, we present a prototype system that implements the two proposed methods for FER and MER as a tool that allows to analyze videos.

Joint Deep Learning of Facial Expression Synthesis and Recognition

Recently, deep learning based facial expression recognition (FER) methods have attracted considerable attention and they usually require large-scale labelled training data. Nonetheless, the publicly available facial expression databases typically contain a small amount of labelled data. In this paper, to overcome the above issue, we propose a novel joint deep learning of facial expression synthesis and recognition method for effective FER. More specifically, the proposed method involves a two-stage learning procedure. Firstly, a facial expression synthesis generative adversarial network (FESGAN) is pre-trained to generate facial images with different facial expressions. To increase the diversity of the training images, FESGAN is elaborately designed to generate images with new identities from a prior distribution. Secondly, an expression recognition network is jointly learned with the pre-trained FESGAN in a unified framework. In particular, the classification loss computed from the recognition network is used to simultaneously optimize the performance of both the recognition network and the generator of FESGAN. Moreover, in order to alleviate the problem of data bias between the real images and the synthetic images, we propose an intra-class loss with a novel real data-guided back-propagation (RDBP) algorithm to reduce the intra-class variations of images from the same class, which can significantly improve the final performance. Extensive experimental results on public facial expression databases demonstrate the superiority of the proposed method compared with several state-of-the-art FER methods.

Expression Recognition Method Based on a Lightweight Convolutional Neural Network

Effective emotion recognition algorithms can help machines better understand people and promote the development of human-computer interaction applications. In recent years, many research efforts have used benchmark expression data to train deep neural network models to achieve state-of-art results. These high-accuracy models usually contain hundreds of layers, so they require complex calculations and may not be suitable for real-world scenarios. This paper proposes a lightweight emotion recognition (LER) model to handle the latency problem under natural conditions. The three main contributions of this paper are as follows. 1) The LER model incorporates a densely connected convolution layer and model compression techniques into a framework that eliminates redundancy parameters. 2) Multichannel input is introduced in our work to preprocess the image data, which improves the learning ability of the model. 3) Experiments show that the proposed LER model has better performance on the FER2013 and FERPLUS datasets compared with other lightweight models. Compared with the VGG13 used in previous work, the LER model achieves higher accuracy and reduces the number of parameters by 97 times. Finally, the FERFIN dataset is created, which had fewer noise data and more accurate labels than the FERPLUS dataset.