Facial Expression Classification Research Articles

Intelligent facial expression recognition and classification using optimal deep transfer learning model

Facial expression is commonly utilized by humans to deliver their mood and emotional state to other people. Facial expression recognition (FER) becomes a hot research area in recent days, and it is a tedious process owing to the presence of high intra-class variation. The conventional methods for FEC are mainly based on handcrafted features with a classification model trained on image or video datasets. Since the facial datasets involve large variations in the images and comprise partial faces, it is needed to design automated FER models. The latest advancements in artificial intelligence (AI) and deep learning (DL) models find useful for better understanding of facial emotions related to face images. In this aspect, this paper presents an intelligent FER using optimal deep transfer learning (IFER-DTFL) model. The proposed IFER-DTFL technique aims to detect the face and identify the facial expressions automatically. The IFER-DTFL technique encompasses a three state process: face detection, feature extraction, and expression classification. In addition, a mask RCNN model is used for the detection of faces. Moreover, the Adam optimizer with Densely Connected Networks (DenseNet121) model is employed for feature extraction process. Furthermore, the weighted kernel extreme learning machine (WKELM) model is utilized to classify the facial expressions. A comprehensive set of simulations were carried out on benchmark dataset and the results are inspected under varying aspects. The experimental results pointed out the supremacy of the IFER-DTFL technique over the other recent techniques interms of several performance measures. • Propose a deep learning based facial expression recognition technique. • Apply Mask RCNN model for face detection process. • Employ Adam optimizer with DenseNet and WKELM model for recognition.

Deep hybrid learning for facial expression binary classifications and predictions

• The Deep Hybrid Learning (DHL) Model is developed for facial expression classification. • Combined architecture with CNN, LSTM, and Transfer Learning are used. • The dropout layers are used to avoid higher computation time. • Different Epoch sizes are used to train the model. • Results are compared with existing models using accuracy, precision, recall, and f1-score. Image processing is a technique used for applying different operations to an image to produce an improved image or extract relevant information. Image processing has multiple applications in numerous fields, such as robotics, vision, pattern recognition, video processing, and the medical industry. One prominent application of facial recognition in image processing is identifying human expression. This research examines the accuracy of categorizing human facial expressions as happy or angry with deep learning and transfer learning methods such as CNN, LSTM, Inception, ResNet, VGG, Xception, and InceptionResnet. The proposed deep hybrid learning (DHL) approach classifies facial expressions using transfer learning and deep neural networks. This approach emphasizes the enhancement of prediction and classification by combining multiple deep learning models to perform better than a single model. The proposed model has a testing accuracy of 81.42% and a training accuracy of 95.93% with a multisource image dataset.

PC-FACS October 2022.

Background: Facial mimicry and emotion recognition are socio-cognitive abilities involved in adaptive socio-emotional behavior.1,2 Are mu-opioid receptors (MORs; key affiliation and social bonding mediators) involved in emotional facial expression categorization and spontaneous mimicry?

The time course of categorical perception of facial expressions

Decoding emotions on others’ faces is one of the most important functions of the human brain, which has been widely studied in cognitive neuroscience. However, the precise time course of facial expression categorization in the human brain is still a matter of debate. Here we used an original paradigm to measure categorical perception of facial expression changes during event-related potentials (ERPs) recording, in which a face stimulus dynamically switched either to a different expression (between-category condition) or to the same expression (within-category condition), the physical distance between the two successive faces being equal across conditions. The switch between faces generated a negative differential potential peaking at around 160 ms over occipito-temporal regions, similar in term of latency and topography to the well-known face-selective N170 component. This response was larger in the condition where the switch occurred between faces that were perceived as having different facial expressions compared to the same expression. In addition, happy expressions were categorized around 20 ms faster than fearful expressions (respectively, 135 and 156 ms). These findings provide evidence that changes of facial expressions are categorically perceived as early as 160 ms following stimulus onset over the occipito-temporal cortex.

Arousal-driven interactions between reward motivation and categorization of emotional facial expressions.

Reward motivation and emotion share common dimensions of valence and arousal, but the nature of interactions between the two constructs is relatively unclear. On the one hand, based on the common valence dimension, valence-compatible interactions are expected where reward motivation would facilitate the processing of compatible (i.e., positive) emotion and hamper the processing of incompatible (i.e., negative) emotion. On the other hand, one could hypothesize valence-general interactions driven by the arousal dimension, where the processing of both positive and negative emotions would be facilitated under reward motivation. Currently, the evidence for valence-compatible vs. valence-general type interactions between reward motivation and goal-relevant emotion is relatively mixed. Moreover, as most of the previous work focused primarily on appetitive motivation, the influence of aversive motivation on goal-relevant emotion is largely unexplored. To address these important gaps, in the present study, we investigated the interactions between motivation and categorization of facial emotional expressions by manipulating the valence dimension of motivation (appetitive and aversive motivation levels) together with that of emotion (positive and negative valence stimuli). Specifically, we conducted two behavioral experiments to separately probe the influence of appetitive and aversive motivation (manipulated via an advance cue signaling the prospect of monetary gains in Experiment 1 and losses in Experiment 2, respectively) on the categorization of happy, fearful, and neutral faces. We tested the two competing hypotheses regarding the interactions between appetitive/aversive motivation and emotional face categorization: Valence-compatible vs. Valence-general. We found evidence consistent with valence-general interactions where both appetitive and aversive motivation facilitated the categorization of happy and fearful faces relative to the neutral ones. Our findings demonstrate that interactions between reward motivation and categorization of emotional faces are driven by the arousal dimension, not by valence.

Multi-Feature Integrated Concurrent Neural Network for Human Facial Expression Recognition

<p>Facial expression helps to communicate between the people for conveying abundant information about human emotions. Facial expression classification is applied in various fields such as remote learning education, medical care, and smart traffic. However, due to the complexity and diversity of the facial emotions, the present facial expression recognition model causes a low recognition rate and it is hard to extract the precise features that are related to facial expression changes. In order to overcome this problem, we proposed Multi-feature Integrated Concurrent Neural Network (MICNN) which is significantly different from the single neural network architectures. It aggregates the prominent features of facial expressions by integrating the three kinds of networks such as Sequential Convolutional Neural Network (SCNN), Residual Dense Network (RDN), and Attention Residual Learning Network (ARLN) to enhance the accuracy rate of facial emotions detection system. Additionally, Local Binary Pattern (LBP) and Principal Component Analysis (PCA) are applied for representing the facial features and these features are combined with the texture features identified by the Gray-level Co-occurrence Matrix (GLCM). Finally, the integrated features are fed into softmax layer to classify the facial images. The experiments are carried out on benchmark datasets by applying k-fold cross-validation and the results demonstrate the superiority of the proposed model.</p> <p> </p>

Relation and context augmentation network for facial expression recognition

Facial Expression Recognition (FER) is a challenging task due to the complex properties of human facial expression. Recently, convolutional neural networks (CNNs) have been widely adopted by most FER approaches. However, CNN-models extract features by using convolutional and pooling operations which ignore the relations between pixels and channels. The relations among spatial positions and channels provide crucial information which can be leveraged for facial expression classification. Another important aspect of FER is utilization of global and local contextual information to improve recognition performance. In this work, we present a deep network, the Relation and Context Augmentation Network (RCANet), for facial expression classification. RCANet consists of two relation modules and a context module. The relation modules compute global relations in spatial and channel dimensions. The context module is composed of cascaded context units to capture multi-scale contextual information. Extensive experiments are conducted on two popular in-the-wild FER datasets, including RAF-DB and AffectNet. Experimental results demonstrate that our proposed method achieves 90.15% and 65.65% accuracy rate on the RAF-DB and AffectNet datasets respectively. • We propose a relation and context augmentation network which constructs global relations for facial expression recognition. • The spatial relation module and the channel relation module encode image-level spatial and channel relations. • The cascaded context augmentation module facilitates feature reuse and generates feature maps with more spatial details.

Emotional Intelligence in Humanoid Robots: Social and Technical Applications

Main challenge in the human–robot interaction is to endow the robots with emotional intelligence making the interaction more genuine and natural. Main aspect in achieving this goal is the robot’s capability to interpret human emotions. Destiny humanoid robot will be one of the most widely used agents in human-robot interaction. Destiny robot will be equipped with a module of facial expression recognition based on CNN (convolutional neural network) facial expression recognition model with high accuracy and capability. This module will be integrated into Destiny's SDK that will allow facial expression classification. Another approach is developing a tool for emotion recognition through vocal features, such as tone, timbre, pitch or loudness making it possible to acquire machine intelligence with emotional intelligence. This feature would be especially helpful for humans suffering from emotional disorders or from the inability to express emotions. The model can detect emotional signals in the human voice using CNN.

Classification of Facial Expressions using Convolutional Neural Networks

We can recognize the emotion of a human by seeing their facial expression and it is an efficient way of human communication. It is the easiest way and essential technology for realizing the human and machine interaction. Facial expression recognition task can be able to classify the face images into various categories of emotions such as happy, sad, angry, fear, surprise, disgust and neutral. In this paper, we are analysing and efficiently classifying each facial image into one of the emotion category. There are numerous approaches to address and solve this problem, out of them convolutional neural network (CNN) is the best approach. Here, we are proposing a novel technique called facial emotion recognition using convolutional neural networks. It is based on the feature extractor to extract the feature and the classifier to produce the label based on the feature. The extraction of feature may be imprecise by variance of location of object and lighting condition on the image. The feature of image can be extracted without user defined feature engineering, and classifier model is integrated with feature extractor to produce the result when input is given. In this way, the CNN approach can produces a feature location invariant image classifier that achieves higher accuracy than conventional linear classifier and our model classified the emotions with 66.62 accuracy.

Uncertain and biased facial expression recognition based on depthwise separable convolutional neural network with embedded attention mechanism

Datasets play a crucial role in the development of facial expression recognition (FER), but most of the datasets suffer from obvious uncertainties and biases caused by different cultures and collection conditions. To look deeper into these issues, this paper first conducts two sets of experiments, face detection, and facial expression (FE) classification. They are based on three datasets (CK+, FER2013, and RAF–DB), which are collected from lab and wild environments. This paper proposes a network, depthwise separable convolutional neural network (CNN) with an embedded attention mechanism (DSA–CNN) for expression recognition. First, at the preprocessing stage, we obtain the maximum expression range clipping, which is calculated from 81 facial landmark points to filter nonface interferences. Then, we use DSA–CNN, which is based on a coordinate squeeze-and-excitation (CSE) attention for feature extraction. Finally, to further deal with imbalanced class biases and uncertainties issues, this paper proposes a class-weighted cross-entropy loss (CCE-loss) to alleviate the imbalance among seven emotional classes. Then, we combine CCE-loss with ranking regularization loss (RR-loss) and self-importance weighting cross-entropy loss (SCE-loss) at label amend stage, to jointly guide the training of the network. Extensive experiments on three FER datasets demonstrate that our proposed method outperforms most of the state-of-the-art methods eventually.

The Effect of Layer Batch Normalization and Droupout of CNN model Performance on Facial Expression Classification

One of the implementations of face recognition is facial expression recognition in which a machine can recognize facial expression patterns from the observed data. This study used two models of convolutional neural network, model A and model B. The first model A was without batch normalization and dropout layers, while the second model B used batch normalization and dropout layers. It used an arrangement of 4 layer models with activation of ReLU and Softmax layers as well as 2 fully connected layers for 5 different classes of facial expressions of angry, happy, normal, sad, and shock faces. Research Metodology are 1). Data Analysis, 2). Preprocessing grayscaling, 3). Convolutional Neural Network (CNN), 4). Model validation Testing, Obtained an accuracy of 64.8% for training data and accuracy of 63.3% for validation data. The use of dropout layers and batch normalization could maintain the stability of both training data and validation data so that there was no overfitting. By dividing the batch size on the training data into 50% with 200 iterations, aiming to make the load on each training model lighter, by using the learning rate to be 0.001 which works to improve the weight value, thus making the training model work to be fast without crossing the minimum error limit. Accuracy results in the classification of ekp facial receipts from the distance of the camera to the face object about 30 cm in the room with the use of bright enough lighting by 78%.

Facial Expression Recognition: One Attention-Modulated Contextual Spatial Information Network.

Facial expression recognition (FER) in the wild is a challenging task due to some uncontrolled factors such as occlusion, illumination, and pose variation. The current methods perform well in controlled conditions. However, there are still two issues with the in-the-wild FER task: (i) insufficient descriptions of long-range dependency of expression features in the facial information space and (ii) not finely refining subtle inter-classes distinction from multiple expressions in the wild. To overcome the above issues, an end-to-end model for FER, named attention-modulated contextual spatial information network (ACSI-Net), is presented in this paper, with the manner of embedding coordinate attention (CA) modules into a contextual convolutional residual network (CoResNet). Firstly, CoResNet is constituted by arranging contextual convolution (CoConv) blocks of different levels to integrate facial expression features with long-range dependency, which generates a holistic representation of spatial information on facial expression. Then, the CA modules are inserted into different stages of CoResNet, at each of which the subtle information about facial expression acquired from CoConv blocks is first modulated by the corresponding CA module across channels and spatial locations and then flows into the next layer. Finally, to highlight facial regions related to expression, a CA module located at the end of the whole network, which produces attentional masks to multiply by input feature maps, is utilized to focus on salient regions. Different from other models, the ACSI-Net is capable of exploring intrinsic dependencies between features and yielding a discriminative representation for facial expression classification. Extensive experimental results on AffectNet and RAF_DB datasets demonstrate its effectiveness and competitiveness compared to other FER methods.

Facial expression detection and classification using SVM, CNN and decision tree algorithm

The human face is often used as a visual representation of information, which is why facial expression recognition is very important in terms of human-machine interaction. It can be used for various applications such as detecting mental disorders and understanding human behavior.Despite the advantages of facial expression recognition technology, the high recognition rate to be achieved by a computer is still challenging. Two commonly used methods are geometry and appearance . Machine learning methods like CNN, Decision tree and SVM were applied to identify the human emotions like happiness, fear, disgust, anger, surprise, sadness and neutrality.

Learning inter-class optical flow difference using generative adversarial networks for facial expression recognition

Facial expression recognition is a fine-grained task because different emotions have subtle facial movements. This paper proposes to learn inter-class optical flow difference using generative adversarial networks (GANs) for facial expression recognition. Initially, the proposed method employs a GAN to produce inter-class optical flow images from the difference between the static fully expressive samples and neutral expression samples. Such inter-class optical flow difference is used to highlight the displacement of facial parts between the neutral facial images and fully expressive facial images, which can avoid the disadvantage that the optical flow change between adjacent frames of the same video expression image is not obvious. Then, the proposed method designs four-channel convolutional neural networks (CNNs) to learn high-level optical flow features from the produced inter-class optical flow images, and high-level static appearance features from the fully expressive facial images, respectively. Finally, a decision-level fusion strategy is adopted to implement facial expression classification. The proposed method is validated on two public facial expression databases, BAUM_1a, SAMM and AFEW5.0, demonstrating its promising performance.

A novel multi-feature fusion deep neural network using HOG and VGG-Face for facial expression classification

A novel multi-feature fusion deep neural network using HOG and VGG-Face for facial expression classification

An Improved Facial Expression Recognition Method Using Combined Hog and Gabor Features

Lately, face recognition technology has been a significant study and a topic for generations. It remains a difficult task because of the variability of wide interclass. The subject of facial expression recognition is addressed in this research using a practical method. This method can recognize the human face and it is various features such as the eyes, brows, and lips. The motions or deformations of the face muscles are the cause of facial expressions. In addition, computer vision tasks such as texture recognition and categorization are commonly used. Furthermore, feature extraction basically discovers groups of features that demonstrate an image of visual texture. It is a critical phase to complete the operation. This work extracts features utilizing Histogram of Oriented Gradients (HOG) and Gabor approaches and then combines extracted features to improve the accuracy of facial expression detection. The derived features were particularly sensitive to object deformations. Later on, the classification of facial expression is handled using (Support Vector Machine) SVM. Analyze the proposed approach on FER 2013 data to see how well it performs. The proposal has a categorization rate of 63.82% on average. The proposed technique determines the comparable classification accuracy as shown in experimental findings. To improve this work it is planned to use deep features and combined them with HOG or Gabor, as well as to show the efficiency of the work it can be implemented with more datasets such as the JAFFE database.

Contextualising facial expressions: The effect of temporal context and individual differences on classification.

The influence of context on facial expression classification is most often investigated using simple cues in static faces portraying basic expressions with a fixed emotional intensity. We examined (1) whether a perceptually rich, dynamic audiovisual context, presented in the form of movie clips (to achieve closer resemblance to real life), affected the subsequent classification of dynamic basic (happy) and non-basic (sarcastic) facial expressions and (2) whether people's susceptibility to contextual cues was related to their ability to classify facial expressions viewed in isolation. Participants classified facial expressions-gradually progressing from neutral to happy/sarcastic in increasing intensity-that followed movie clips. Classification was relatively more accurate and faster when the preceding context predicted the upcoming expression, compared with when the context did not. Speeded classifications suggested that predictive contexts reduced the emotional intensity required to be accurately classified. More importantly, we show for the first time that participants' accuracy in classifying expressions without an informative context correlated with the magnitude of the contextual effects experienced by them-poor classifiers of isolated expressions were more susceptible to a predictive context. Our findings support the emerging view that contextual cues and individual differences must be considered when explaining mechanisms underlying facial expression classification.

Facial Motion Analysis beyond Emotional Expressions.

Facial motion analysis is a research field with many practical applications, and has been strongly developed in the last years. However, most effort has been focused on the recognition of basic facial expressions of emotion and neglects the analysis of facial motions related to non-verbal communication signals. This paper focuses on the classification of facial expressions that are of the utmost importance in sign languages (Grammatical Facial Expressions) but also present in expressive spoken language. We have collected a dataset of Spanish Sign Language sentences and extracted the intervals for three types of Grammatical Facial Expressions: negation, closed queries and open queries. A study of several deep learning models using different input features on the collected dataset (LSE_GFE) and an external dataset (BUHMAP) shows that GFEs can be learned reliably with Graph Convolutional Networks simply fed with face landmarks.

Effects of the mu-opioid receptor agonist morphine on facial mimicry and emotion recognition

Facial mimicry and emotion recognition are two socio-cognitive abilities involved in adaptive socio-emotional behavior, promoting affiliation and the establishment of social bonds. The mu-opioid receptor (MOR) system plays a key role in affiliation and social bonding. However, it remains unclear whether MORs are involved in the categorization and spontaneous mimicry of emotional facial expressions. Using a randomized, placebo-controlled, double-blind, between-subjects design, we investigated in 82 healthy female volunteers the effects of the specific MOR agonist morphine on the recognition accuracy of emotional faces (happiness, anger, fear), and on their facial mimicry (measured with electromyography). Frequentist statistics did not reveal any significant effects of drug administration on facial mimicry or emotion recognition abilities. However, post hoc Bayesian analyses provided support for an effect of morphine on facial mimicry of fearful facial expressions. Specifically, compared to placebo, morphine reduced mimicry of fear, as shown by lower activity of the frontalis muscle. Bayesian analyses also provided support for the absence of a drug effect on mimicry of happy and angry facial expressions, which were assessed with the zygomaticus major and corrugator supercilii muscles, as well as on emotion recognition accuracy. These findings suggest that MOR activity is involved in automatic facial responses to fearful stimuli, but not in their identification. Overall, the current results, together with the previously reported small effects of opioid compounds, suggest a relatively marginal role of the MOR system in emotion simulation and perception.

HUMAN FACIAL EMOTION RECOGNITION USING CNN

Human beings express emotions in everyday interactions. Understanding their emotions and knowing how to react to these expressions greatly enhances the interaction. An automatic Facial Expression Recognition system must try to solve the following problems: detection and location of faces ina cluttered scene, facial feature extraction, and facial expression classification. Knowing the user emotion, the system can adapt to the user. Facial expressions play an important role in recognition of emotions and are used in the process of non-verbal communication. They are very important in daily emotional communication, just next to the tone of voice. They are also an indicator of feelings, allowing a man to express an emotional state. The main motivation behind this project is to detect mental health of an individual.