Human Facial Expression Recognition Research Articles

A CNN-improved and channel-weighted lightweight human facial expression recognition method

A CNN-improved and channel-weighted lightweight human facial expression recognition method

Spatio-temporal features based human facial expression recognition

Spatio-temporal features based human facial expression recognition

Video sequence-based human facial expression recognition using Transformer networks

Video sequence-based human facial expression recognition using Transformer networks

Recognition of Human Facial Expressions Using DCT-DWT and Artificial Neural Network

Facial expressions are a term that expresses a group of movements of the facial fore muscles that is related to one's own human emotions. Human–computer interaction (HCI) has been considered as one of the most attractive and fastest-growing fields. Adding emotional expression’s recognition to expect the users’ feelings and emotional state can drastically improves HCI. This paper aims to demonstrate the three most important facial expressions (happiness, sadness, and surprise). It contains three stages; first, the preprocessing stage was performed to enhance the facial images. Second, the feature extraction stage depended on Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT) methods. Third, the recognition stage was applied using an artificial neural network, known as Back Propagation Neural Network (BPNN), on database images from Cohen-Kanade. The method was shown to be very efficient, where the total rate of recognition of the three facial expressions was 92.9%.

An Effective Combination of Textures and Wavelet Features for Facial Expression Recognition

In order to explore the accompanying examination goals for facial expression recognition, a proper combination of classification and adequate feature extraction is necessary. If inadequate features are used, even the best classifier could fail to achieve accurate recognition. In this paper, a new fusion technique for human facial expression recognition is used to accurately recognize human facial expressions. A combination of Discrete Wavelet Features (DWT), Local Binary Pattern (LBP), and Histogram of Gradients (HoG) feature extraction techniques was used to investigate six human emotions. K-Nearest Neighbors (KNN), Decision Tree (DT), Multi-Layer Perceptron (MLP), and Random Forest (RF) were chosen for classification. These algorithms were implemented and tested on the Static Facial Expression in Wild (SWEW) dataset which consists of facial expressions of high accuracy. The proposed algorithm exhibited 87% accuracy which is higher than the accuracy of the individual algorithms.

Towards Machine Recognition of Facial Expressions of Pain in Horses.

Simple SummaryFacial activity can convey valid information about the experience of pain in a horse. However, scoring of pain in horses based on facial activity is still in its infancy and accurate scoring can only be performed by trained assessors. Pain in humans can now be recognized reliably from video footage of faces, using computer vision and machine learning. We examine the hurdles in applying these technologies to horses and suggest two general approaches to automatic horse pain recognition. The first approach involves automatically detecting objectively defined facial expression aspects that do not involve any human judgment of what the expression “means”. Automated classification of pain expressions can then be done according to a rule-based system since the facial expression aspects are defined with this information in mind. The other involves training very flexible machine learning methods with raw videos of horses with known true pain status. The upside of this approach is that the system has access to all the information in the video without engineered intermediate methods that have filtered out most of the variation. However, a large challenge is that large datasets with reliable pain annotation are required. We have obtained promising results from both approaches.Automated recognition of human facial expressions of pain and emotions is to a certain degree a solved problem, using approaches based on computer vision and machine learning. However, the application of such methods to horses has proven difficult. Major barriers are the lack of sufficiently large, annotated databases for horses and difficulties in obtaining correct classifications of pain because horses are non-verbal. This review describes our work to overcome these barriers, using two different approaches. One involves the use of a manual, but relatively objective, classification system for facial activity (Facial Action Coding System), where data are analyzed for pain expressions after coding using machine learning principles. We have devised tools that can aid manual labeling by identifying the faces and facial keypoints of horses. This approach provides promising results in the automated recognition of facial action units from images. The second approach, recurrent neural network end-to-end learning, requires less extraction of features and representations from the video but instead depends on large volumes of video data with ground truth. Our preliminary results suggest clearly that dynamics are important for pain recognition and show that combinations of recurrent neural networks can classify experimental pain in a small number of horses better than human raters.

Action unit classification for facial expression recognition using active learning and SVM

Automatic facial expression analysis remains challenging due to its low recognition accuracy and poor robustness. In this study, we utilized active learning and support vector machine (SVM) algorithms to classify facial action units (AU) for human facial expression recognition. Active learning was used to detect the targeted facial expression AUs, while an SVM was utilized to classify different AUs and ultimately map them to their corresponding facial expressions. Active learning reduces the number of non-support vectors in the training sample set and shortens the labeling and training times without affecting the performance of the classifier, thereby reducing the cost of labeling samples and improving the training speed. Experimental results show that the proposed algorithm can effectively suppress correlated noise and achieve higher recognition rates than principal component analysis and a human observer on seven different facial expressions.

Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom

The automatical recognition of human facial expression has attracted attention in the field of computer vision and machine learning. Previous works on this topic set many constraints, such as the impact caused by restricted scenarios and low image quality. To address those problems, we propose a new infrared facial expression recognition method with multi-label distribution learning for understanding non-verbal behaviors in the classroom. Specifically, we first compute the feature similarities of seven basic facial expressions to describe the relationship among the adjacent expression images. Then, the similarity values are fitted by a Cauchy distribution function. Furthermore, we construct a new deep network with Cauchy distribution-based label learning (CDLLNet), instead of the conventional single expression labels. By these revised labels, one infrared facial expression can contribute to the learning of neighboring expression labels, as well as its real expression label. The performance of proposed network is evaluated on two facial expression datasets: Oulu-CASIA and CK+. Several qualitative and quantitative experimental results verify that the CDLLNet network can achieve robust results and significantly outperforms the existing state-of-the-art facial expression algorithms.

Regional Attention Network (RAN) for Head Pose and Fine-Grained Gesture Recognition

Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose "informativeness" can be evaluated by the attention mechanism for inferring fine-grained gestures/actions. To this end, we propose a novel end-to-end \textbf{Regional Attention Network (RAN)}, which is a fully Convolutional Neural Network (CNN) to combine multiple contextual regions through attention mechanism, focusing on parts of the images that are most relevant to a given task. Our regions consist of one or more consecutive cells and are adapted from the strategies used in computing HOG (Histogram of Oriented Gradient) descriptor. The model is extensively evaluated on ten datasets belonging to 3 different scenarios: 1) head pose recognition, 2) drivers state recognition, and 3) human action and facial expression recognition. The proposed approach outperforms the state-of-the-art by a considerable margin in different metrics.

Deep learning based facial expression recognition using improved Cat Swarm Optimization

Human emotional facial expressions play a vital role in interpersonal relations. Automated facial expression recognition has always remained a challenging problem in real-life applications as people vary significantly in the way of showing their expressions. Recently various approaches have been proposed for automatically analyzing the facial expression of a person. In this paper, a novel approach to human facial expression recognition by applying a modified version of the Cat Swarm Optimization (CSO) algorithm, called Improved Cat Swarm Optimization (ICSO) algorithm is proposed. The input image given to the proposed system retrieves similar images from the dataset as well as identifies the person’s emotional state through facial expressions. Deep features present in the face image are extracted using Deep Convolution Neural Network (DCNN) approach. ICSO is proposed to select optimal features from the face image that can uniquely distinguish the facial expression of a person. Employing DCNN with ICSO improves the retrieval performance of the proposed system. Ensemble classifiers that employ Neural Network (NN) and Support Vector Machine (SVM) are implemented to classify facial expressions such as normal, happy, sad, surprise, fear and angry. The performance of the proposed system is evaluated using JAFFE, CK+, Pie datasets and some real-world images. The proposed system outperforms the existing system, thus achieving superior accuracy and reduced computation time.

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

Owing to the development and advancement of artificial intelligence, numerous works were established in the human facial expression recognition system. Meanwhile, the detection and classification of micro-expressions are attracting attentions from various research communities in the recent few years. In this paper, we first review the processes of a conventional optical-flow-based recognition system, which comprised of facial landmarks annotations, optical flow guided images computation, features extraction and emotion class categorization. Secondly, a few approaches have been proposed to improve the feature extraction part, such as exploiting GAN to generate more image samples. Particularly, several variations of optical flow are computed in order to generate optimal images to lead to high recognition accuracy. Next, GAN, a combination of Generator and Discriminator, is utilized to generate new "fake" images to increase the sample size. Thirdly, a modified state-of-the-art Convolutional neural networks is proposed. To verify the effectiveness of the the proposed method, the results are evaluated on spontaneous micro-expression databases, namely SMIC, CASME II and SAMM. Both the F1-score and accuracy performance metrics are reported in this paper.

PCA + LDA Fuzzy Based Model for Emotional Nature Recognition of Human Video

Human expresses their emotions by means of verbal and nonverbal communication. Nonverbal communications are done mainly using facial expression. This paper aims to recognize human emotion using nonverbal communication of human facial expressions. Different mathematical techniques like: principle component analysis (PCA), linear discriminate analysis (LDA) and independent component analysis (ICA) are widely used for human facial expression recognition. This paper applied fusion of PCA and LDA based model for facial video emotion recognition with neural network (NN), fuzzy approach and Ekman’s proposed concept of action units of faces. Moreover, results obtained in linguistic form using action units with fuzzy approach on unknwn individual persons for identification of nature of input video and compare with the actual data to validate the model. This paper concludes that developed approach provides 99% accuracy for human facial expression recognition and identification of nature of input video.

Facial expression recognition techniques: a comprehensive survey

Over the past decades, facial expression recognition (FER) has become an interesting research area and achieved substantial progress in computer vision. FER is to detect human emotional state related to biometric traits. Developing a machine based human FER system is a quite challenging task. Various FER systems are developed by analysing facial muscle motion and skin deformation based algorithms. In conventional FER system, the developed algorithms work on the constrained database. In the unconstrained environment, the efficacy of existing algorithms is limited due to certain issues during image acquisition. This study presents a detailed study on FER techniques, classifiers and datasets used for analysing the efficacy of the recognition techniques. Moreover, this survey will assist researchers in understanding the strategies and innovative methods that address the issues in a real-time application. Finally, the review presents the challenges encountered by FER system along with the future direction.

Development of an Automated Pain Facial Expression Detection System for Sheep (Ovis Aries).

Simple SummaryDetecting signs of pain in sheep is a challenging problem, as they are a prey species and would usually try to hide any signs that they are unwell or injured. This means that treating ill or injured sheep and preventing any further spread of contagious diseases such as footrot can be slow. The recent development and publication of a Sheep Pain Facial Expression Scale (SPFES) has provided a tool to reliably detect pain in this species. However, due to the increase in intensification in farming and larger flock sizes being cared for by individual farmers, there is less time to spend monitoring sheep for changes in behaviour that may indicate illness or injury. Having an automated system that could detect changes in the facial expression of individual sheep would mean that farmers could receive information directly about particular individuals that need assessment. This would allow treatment to be provided in a timely and direct manner, reducing suffering. We have been developing the SPFES further in order for it to become an automated system. In this paper, we present our novel framework that integrates SPFES concepts with automatic facial expression analysis technologies.The use of technology to optimize the production and management of each individual animal is becoming key to good farming. There is a need for the real-time systematic detection and control of disease in animals in order to limit the impact on animal welfare and food supply. Diseases such as footrot and mastitis cause significant pain in sheep, and so early detection is vital to ensuring effective treatment and preventing the spread across the flock. Facial expression scoring to assess pain in humans and non-humans is now well utilized, and the Sheep Pain Facial Expression Scale (SPFES) is a tool that can reliably detect pain in this species. The SPFES currently requires manual scoring, leaving it open to observer bias, and it is also time-consuming. The ability of a computer to automatically detect and direct a producer as to where assessment and treatment are needed would increase the chances of controlling the spread of disease. It would also aid in the prevention of resistance across the individual, farm, and landscape at both national and international levels. In this paper, we present our framework for an integrated novel system based on techniques originally applied for human facial expression recognition that could be implemented at the farm level. To the authors’ knowledge, this is the first time that this technology has been applied to sheep to assess pain.

FEATURE VECTOR GENERATION FOR THE FACIAL EXPRESSION RECOGNITION USING NEO-FUZZY SYSTEM

Context. The article is devoted to the problem of a training data set forming for the automatic human emotions recognition system on the basis of a multidimensional extended neo-fuzzy neuron. The aspects of choice the attributes vector’s dimension and composition, their influence on the system learning rate are considered. The object of research is the method of multidimensional data clustering. The subject of research is two-dimensional images geometric features systematization. Objective. The main goal of the work is to develop an approach to person’s face expression description using geometric features fixed set that can be obtained by video sequence frames processing. Method. To study the facial expressions recognition system it is proposed to form a feature vector consisting of characteristic points coordinates. There were selected points that relate to the location and shape of the eyelids, eyebrows, eye pupils, lips contours, nose wings, nasolabial folds. Such points can be easily found during the automatic image processing using known contour detectors. Also, the possibility of using for the human facial expression description not the coordinates of characteristic points, but the distances between them, was investigated. From these distances a different feature vector was created, the properties of which were compared with the points coordinates vector. Results. The developed recognition system on the basis of a multidimensional extended neo-fuzzy neuron have been implemented in software and investigated for solving the problem of facial expression classification. A comparison between the attribute vectors that are different in composition and dimension is made. The structure for the feature vector, which provides high system learning rate, and does not require the additional structural elements was chosen. Conclusions. The experimental study fully confirms the effectiveness of the developed approach for the human facial expressions recognition using a multidimensional extended neo-fuzzy neuron.

Multidimensional Bernstein polynomials and Bezier curves: Analysis of machine learning algorithm for facial expression recognition based on curvature

In this paper, by using partial derivative formulas of generating functions for the multidimensional unification of the Bernstein basis functions and their functional equations, we derive derivative formulas and identities for these basis functions and their generating functions. We also give a conjecture and some open questions related to not only subdivision property of these basis functions, but also solutions of a higher-order special differential equations. Moreover, we provide an implementation for a real world problem of human facial expression recognition with the help of curvature of Bezier curves whose machine learning supported by statistical evaluations on feature vectors using in the aforementioned machine learning algorithm.

Mixture Statistic Metric Learning for Robust Human Action and Expression Recognition

Background objects and textures in real-world video sequences often pose great challenges for human action and facial expression recognition. This paper proposes a mixture statistic metric learning for recognizing human actions and facial expressions in realistic “in the wild” scenarios. In the proposed method, multiple statistics, including temporal means and covariance matrices, as well as parameters of spatial Gaussian mixture distributions, are explicitly mapped to or generated on symmetric positive definite Riemannian manifolds. An implicit mixture of Mahalanobis metrics is learned from the Riemannian manifolds. The learned metrics place similar pairs in local neighborhoods and dissimilar pairs in relatively orthogonal regions on a regularized manifold. The proposed metric learning method also explores the prior distributions within the multiple statistics in the video sequences. The proposed method is tested on five action video data sets and three facial expression data sets and is compared with various state-of-the-art methods. Recognition accuracy and computational efficiency are evaluated in terms of average recognition rates and computational times in seconds, respectively. Competitive performances achieved on both action and facial expression recognition tasks demonstrate the effectiveness of the proposed method.

Multivariate Pattern Classification of Facial Expressions Based on Large-Scale Functional Connectivity.

It is an important question how human beings achieve efficient recognition of others’ facial expressions in cognitive neuroscience, and it has been identified that specific cortical regions show preferential activation to facial expressions in previous studies. However, the potential contributions of the connectivity patterns in the processing of facial expressions remained unclear. The present functional magnetic resonance imaging (fMRI) study explored whether facial expressions could be decoded from the functional connectivity (FC) patterns using multivariate pattern analysis combined with machine learning algorithms (fcMVPA). We employed a block design experiment and collected neural activities while participants viewed facial expressions of six basic emotions (anger, disgust, fear, joy, sadness, and surprise). Both static and dynamic expression stimuli were included in our study. A behavioral experiment after scanning confirmed the validity of the facial stimuli presented during the fMRI experiment with classification accuracies and emotional intensities. We obtained whole-brain FC patterns for each facial expression and found that both static and dynamic facial expressions could be successfully decoded from the FC patterns. Moreover, we identified the expression-discriminative networks for the static and dynamic facial expressions, which span beyond the conventional face-selective areas. Overall, these results reveal that large-scale FC patterns may also contain rich expression information to accurately decode facial expressions, suggesting a novel mechanism, which includes general interactions between distributed brain regions, and that contributes to the human facial expression recognition.

3D facial expression recognition using kernel methods on Riemannian manifold

Automatic human Facial Expressions Recognition (FER) is becoming of increased interest. FER finds its applications in many emerging areas such as affective computing and intelligent human computer interaction. Most of the existing work on FER has been done using 2D data which suffers from inherent problems of illumination changes and pose variations. With the development of 3D image capturing technologies, the acquisition of 3D data is becoming a more feasible task. The 3D data brings a more effective solution in addressing the issues raised by its 2D counterpart. State-of-the-art 3D FER methods are often based on a single descriptor which may fail to handle the large inter-class and intra-class variability of the human facial expressions. In this work, we explore, for the first time, the usage of covariance matrices of descriptors, instead of the descriptors themselves, in 3D FER. Since covariance matrices are elements of the non-linear manifold of Symmetric Positive Definite (SPD) matrices, we particularly look at the application of manifold-based classification to the problem of 3D FER. We evaluate the performance of the proposed framework on the BU-3DFE and the Bosphorus datasets, and demonstrate its superiority compared to the state-of-the-art methods.

The role of motion and intensity in deaf children’s recognition of real human facial expressions of emotion

ABSTRACTThere is substantial evidence to suggest that deafness is associated with delays in emotion understanding, which has been attributed to delays in language acquisition and opportunities to converse. However, studies addressing the ability to recognise facial expressions of emotion have produced equivocal findings. The two experiments presented here attempt to clarify emotion recognition in deaf children by considering two aspects: the role of motion and the role of intensity in deaf children’s emotion recognition. In Study 1, 26 deaf children were compared to 26 age-matched hearing controls on a computerised facial emotion recognition task involving static and dynamic expressions of 6 emotions. Eighteen of the deaf and 18 age-matched hearing controls additionally took part in Study 2, involving the presentation of the same 6 emotions at varying intensities. Study 1 showed that deaf children’s emotion recognition was better in the dynamic rather than static condition, whereas the hearing children showed no difference in performance between the two conditions. In Study 2, the deaf children performed no differently from the hearing controls, showing improved recognition rates with increasing rates of intensity. With the exception of disgust, no differences in individual emotions were found. These findings highlight the importance of using ecologically valid stimuli to assess emotion recognition.