Facial Expression Recognition Techniques Research Articles

Label distribution learning for compound facial expression recognition in‐the‐wild: A comparative study

AbstractHuman emotional states encompass both basic and compound facial expressions. However, current works primarily focus on basic expressions, consequently neglecting the broad spectrum of human emotions encountered in practical scenarios. Compound facial expressions involve the simultaneous manifestation of multiple emotions on an individual's face. This phenomenon reflects the complexity and richness of human states, where facial features dynamically convey a combination of feelings. This study embarks on a pioneering exploration of Compound Facial Expression Recognition (CFER), with a distinctive emphasis on leveraging the Label Distribution Learning (LDL) paradigm. This strategic application of LDL aims to address the ambiguity and complexity inherent in compound expressions, marking a significant departure from the dominant Single Label Learning (SLL) and Multi‐Label Learning (MLL) paradigms. Within this framework, we rigorously investigate the potential of LDL for a critical challenge in Facial Expression Recognition (FER): recognizing compound facial expressions in uncontrolled environments. We utilize the recently introduced RAF‐CE dataset, meticulously designed for compound expression assessment. By conducting a comprehensive comparative analysis pitting LDL against conventional SLL and MLL approaches on RAF‐CE, we aim to definitively establish LDL's superiority in handling this complex task. Furthermore, we assess the generalizability of LDL models trained on RAF‐CE by evaluating their performance on the EmotioNet and RAF‐DB Compound datasets. This demonstrates their effectiveness without domain adaptation. To solidify these findings, we conduct a comprehensive comparative analysis of 12 cutting‐edge LDL algorithms on RAF‐CE, S‐BU3DFE, and S‐JAFFE datasets, providing valuable insights into the most effective LDL techniques for FER in‐the‐wild.

Emotion knowledge-based fine-grained facial expression recognition

Existing facial expression recognition (FER) techniques rely primarily on seven coarse-grained emotions as emotional labels, which are insufficient to cover the subtle changes in human emotions in the real world. We use 135 fine-grained emotions as emotional benchmarks to address the problem of highly semantically similar fine-grained emotion recognition. In this work, we propose a robust emotion knowledge-based fine-grained (EK-FG) emotion recognition network that captures inter-class relationships and discriminative representations of fine-grained emotions through two prior-based losses: coarse-grained hierarchical loss and fine-grained semantic loss. Specifically, the coarse-grained hierarchical loss obtains a structured semantic representation of fine-grained emotions based on prior knowledge, and captures inter-class relationships through effective category-level push-pull to obtain discriminative representations. The fine-grained semantic loss provides more accurate measurement information for semantic features based on prior knowledge, and enhances the model’s discriminative ability for subtle facial expression differences through regression constraints. Extensive experimental results on the Emo135 dataset demonstrate that EK-FG can effectively overcome the class ambiguity of fine-grained emotion.

CCNN-SVM: Automated Model for Emotion Recognition Based on Custom Convolutional Neural Networks with SVM

The expressions on human faces reveal the emotions we are experiencing internally. Emotion recognition based on facial expression is one of the subfields of social signal processing. It has several applications in different areas, specifically in the interaction between humans and computers. This study presents a simple CCNN-SVM automated model as a viable approach for FER. The model combines a Convolutional Neural Network for feature extraction, certain image preprocessing techniques, and Support Vector Machine (SVM) for classification. Firstly, the input image is preprocessed using face detection, histogram equalization, gamma correction, and resizing techniques. Secondly, the images go through custom single Deep Convolutional Neural Networks (CCNN) to extract deep features. Finally, SVM uses the generated features to perform the classification. The suggested model was trained and tested on four datasets, CK+, JAFFE, KDEF, and FER. These datasets consist of seven primary emotional categories, which encompass anger, disgust, fear, happiness, sadness, surprise, and neutrality for CK+, and include contempt for JAFFE. The model put forward demonstrates commendable performance in comparison to existing facial expression recognition techniques. It achieves an impressive accuracy of 99.3% on the CK+ dataset, 98.4% on the JAFFE dataset, 87.18% on the KDEF dataset, and 88.7% on the FER.

Domestic Cats Facial Expression Recognition Based on Convolutional Neural Networks

Despite extensive research on Facial Expression Recognition (FER) in humans using deep learning technology, significantly less focus has been placed on applying these advancements to recognize facial expressions in domestic animals. Recognizing this gap, our research aims to extend FER techniques specifically to domestic cats, one of the most popular domestic pets. In this paper, we present a real-time system model that employs deep learning to identify and classify cat facial expressions into four categories: Pleased, Angry, Alarmed, and Calm. This innovative model not only helps cat owners understand their pets' behavior more accurately but also holds substantial potential for applications in domestic animal health services. By identifying and interpreting the emotional states of cats, we can address a critical need for improved communication between humans and their pets, fostering better care and well-being for these animals. To develop this system, we conducted extensive experiments and training using a diverse dataset of cat images annotated with corresponding facial expressions. Our approach involved using convolutional neural networks (CNNs) to analyze and learn from the subtleties in feline facial features by investigating the models' robustness considering metrics such as accuracy, precision, recall, confusion matrix, and f1-score. The experimental results demonstrate the high recognition accuracy and practicality of our model, underscoring its effectiveness. This research aims to empower pet owners, veterinarians, and researchers with advanced tools and insights, ensuring the well-being and happiness of domestic cats. Ultimately, our work highlights the potential of FER technology to significantly enhance the quality of life for cats by enabling better understanding and more responsive care from their human companions

Marine predators optimization with deep learning model for video‐based facial expression recognition

AbstractVideo‐based facial expression recognition (VFER) technique intends to categorize an input video into different kinds of emotions. It remains a challenging issue because of the gap between visual features and emotions, problems in handling the delicate movement of muscles, and restricted datasets. One of the effective solutions to solve this problem is the exploitation of efficient features defining facial expressions to carry out FER. Generally, the VFER find useful in several areas like unmanned driving, venue management, urban safety management, and senseless attendance. Recent advances in computer vision and deep learning (DL) techniques enable the design of automated VFER models. In this aspect, this study establishes a new Marine Predators Optimization with Deep Learning Model for Video‐based Facial Expression Recognition (MPODL‐VFER) technique. The presented MPODL‐VFER technique mainly aims to classify different kinds of facial emotions in the video. To accomplish this, the presented MPODL‐VFER technique derives features using the deep convolutional neural network based densely connected network (DenseNet) model. The presented MPODL‐VFER technique employs MPO technique for the hyperparameter adjustment of the DenseNet model. Finally, Elman Neural Network (ENN) model is exploited for emotion recognition purposes. For assuring the enhanced recognition performance of the MPODL‐VFER approach, a comparison study was developed on benchmark dataset. The comprehensive results have shown the significant outcome of MPODL‐VFER model over other approaches.

A gradual self distillation network with adaptive channel attention for facial expression recognition

Facial expression recognition (FER) is widely applied in various real-world applications, such as security, human–computer interaction, and healthcare, leading to high demands for fast and accurate FER techniques. However, it remains a challenging task to design effective and efficient FER techniques that meet the requirements of real-world applications. In this paper, we propose a lightweight Gradual Self Distillation Network with adaptive channel attention (GSDNet) for accurate and efficient FER. Especially, we propose a novel gradual self distillation strategy which enables the network to learn from itself in a gradual and adaptive way. Specifically, the proposed GSDNet consists of a feature extraction backbone with multiple basic blocks. We plug an adaptive classifier after each basic block. Every two neighbor classifiers form “student & teacher” relationship for gradual knowledge distillation. In particular, the gradual self distillation strategy enables the transfer of key knowledge from deep to shallow layers gradually. Besides, an Adaptive Channel Attention Module (ACAM) is designed to enhance the representation capability of each block for adaptively capturing important features and achieve better FER performance. Extensive experiments on three real-world datasets show that the proposed method GSDNet outperforms the baselines, including state-of-the-art methods. Specifically, the accuracy of GSDNet on the RAF-DB, Affect-net, and FERPlus datasets is 90.91%, 66.11%, and 90.32%, separately. The code is available at https://github.com/Emy-cv/GSDNet.

A novel relation‐aware global attention network for sentiment analysis of students in the wisdom classroom

SummaryThe ability of facial expression recognition, the ability to decipher human emotions from facial features, and we can judge the emotional state of human beings by analysing facial expressions. Empowered by this technology, the related research of student sentiment analysis has become a focal point in the realm of educational technology. Teachers can now read the emotional state of their students and estimate the effectiveness of their teaching strategies, enabling them to implement appropriate intervention techniques to enhance teaching outcomes. However, facial expression recognition techniques are currently limited by their shortcomings. Network performance degradation and loss of feature information are key issues that hinder the effectiveness of sentiment analysis of student feedback. To overcome these limitations, in this paper, we propose RFMNet, a novel network model based on deep learning theory. RFMNet, with its higher extraction capability, is designed to accurately analyse student expressions. It introduces a relation‐aware global attention (RGA) module, which facilitates the integration of more discriminative expression features in the image, leading to more refined sentiment analysis. This innovative model also employs a Mish activation function and a focal loss function. The Mish activation function is of particular interest because it helps to avoid the loss of feature information due to neuron deactivation when the ReLU gradient is . This approach results in a more robust and accurate facial expression recognition model. Our model was tested on the FERPlus public dataset. The model achieved an average recognition accuracy of 89.62%, which is a testament to its performance in decoding the facial expressions of students. This precision allows for intelligent processing of teaching information and real‐time feedback, enabling teachers to adapt their teaching strategies to better suit the needs of their students.

Modeling Uncertainty for Low-Resolution Facial Expression Recognition

Recently, facial expression recognition techniques have made significant progress on high-resolution web images. However, in real-world applications, the obtained images are often with low resolution since they are mostly captured in a wide range of public spaces. As a result, the ambiguity of the expression labels hinders recognition performance due to not only subjective emotion annotations but also ambiguous images. Existing approaches tend to perform poorly when the resolution of face images decreases. In this work, we aim to model the aleatoric uncertainty induced by low-image-resolution and label ambiguity for robust facial expression recognition. We propose probabilistic data uncertainty learning to capture the ambiguity induced by poor image resolution. Additionally, we introduce the emotion wheel to learn the label-uncertainty-aware embedding. Moreover, we exploit the ambiguous nature of neutrality and propose a neutral expression constraint to learn more robust features for facial expression recognition. To the best of our knowledge, this is the first work utilizing the intrinsic nature of neutrality as a regularization to benefit model training. Extensive experimental results show the effectiveness and robustness of our approach. Under low-resolution conditions, our proposed method outperforms the state-of-the-art approaches by 3.02% and 3.16% in terms of accuracy on RAF-DB and FERPlus, respectively.

Analyzing the Effect of Diverse Gaze and Head Direction on Facial Expression Recognition With Photo-Reflective Sensors Embedded in a Head-Mounted Display.

As one of the facial expression recognition techniques for Head-Mounted Display (HMD) users, embedded photo-reflective sensors have been used. In this paper, we investigate how gaze and face directions affect facial expression recognition using the embedded photo-reflective sensors. First, we collected a dataset of five facial expressions (Neutral, Happy, Angry, Sad, Surprised) while looking in diverse directions by moving 1) the eyes and 2) the head. Using the dataset, we analyzed the effect of gaze and face directions by constructing facial expression classifiers in five ways and evaluating the classification accuracy of each classifier. The results revealed that the single classifier that learned the data for all gaze points achieved the highest classification performance. Then, we investigated which facial part was affected by the gaze and face direction. The results showed that the gaze directions affected the upper facial parts, while the face directions affected the lower facial parts. In addition, by removing the bias of facial expression reproducibility, we investigated the pure effect of gaze and face directions in three conditions. The results showed that, in terms of gaze direction, building classifiers for each direction significantly improved the classification accuracy. However, in terms of face directions, there were slight differences between the classifier conditions. Our experimental results implied that multiple classifiers corresponding to multiple gaze and face directions improved facial expression recognition accuracy, but collecting the data of the vertical movement of gaze and face is a practical solution to improving facial expression recognition accuracy.

UIBVFEDPlus-Light: Virtual facial expression dataset with lighting.

It is well-known that lighting conditions have an important influence on the automatic recognition of human expressions. Although the impact of lighting on the perception of emotions has been studied in different works, databases of facial expressions do not consider intentional lighting. In this work, a new database of facial expressions performed by virtual characters with four different lighting configurations is presented. This database, named UIBVFEDPlus-Light, is an extension of the previously published UIBVFED virtual facial expression dataset. It includes 100 characters, four lighting configurations and a software application that allows one to interactively visualize the expressions, and manage their intensity and lighting condition. Also, an experience of use is described to show how this work can raise new challenges to facial expression and emotion recognition techniques under usual lighting environments. Thus, opening new study perspectives in this area.

Facial Expression Recognition Techniques and Comparative Analysis Using Classification Algorithms

With the development of technology and hardware possibilities, it has become possible to analyze the changes that occur as a result of the reflection of emotional state on facial expression with computer vision applications. Facial expression analysis systems are used in applications such as security systems, early diagnosis of certain diseases in the medical world, human-computer interaction, safe driving. Facial expression analysis systems developed using image data consist of 3 basic stages. These; extracting the face area from the input data, extracting the feature vectors of the data and classifying the feature vectors. In this study, the features of the dataset were obtained with the AlexNet model, which is one of the deep learning models that achieved successful results in classification problems. In the study in which the comparative analysis of the obtained results is presented, accuracy of 89.7%, 87.8% and 81.7% was obtained with machine learning techniques.

Feature Extraction Techniques for Facial Expression Recognition (FER)

Feature Extraction Techniques for Facial Expression Recognition (FER)

A Multi-View Face Expression Recognition Method Based on DenseNet and GAN

Facial expression recognition (FER) techniques can be widely used in human-computer interaction, intelligent robots, intelligent monitoring, and other domains. Currently, FER methods based on deep learning have become the mainstream schemes. However, these methods have some problems, such as a large number of parameters, difficulty in being applied to embedded processors, and the fact that recognition accuracy is affected by facial deflection. To solve the problem of a large number of parameters, we propose a DSC-DenseNet model, which improves the standard convolution in DenseNet to depthwise separable convolution (DSC). To solve the problem wherein face deflection affects the recognition effect, we propose a posture normalization model based on GAN: a GAN with two local discriminators (LD-GAN) that strengthen the discriminatory abilities of the expression-related local parts, such as the parts related to the eyes, eyebrows, mouth, and nose. These discriminators improve the model’s ability to retain facial expressions and evidently benefits FER. Quantitative and qualitative experimental results on the Fer2013 and KDEF datasets have consistently shown the superiority of our FER method when working with multi-pose face images.

Few-shot learning for facial expression recognition: a comprehensive survey

Facial expression recognition (FER) is utilized in various fields that analyze facial expressions. FER is attracting increasing attention for its role in improving the convenience in human life. It is widely applied in human–computer interaction tasks. However, recently, FER tasks have encountered certain data and training issues. To address these issues in FER, few-shot learning (FSL) has been researched as a new approach. In this paper, we focus on analyzing FER techniques based on FSL and consider the computational complexity and processing time in these models. FSL has been researched as it can solve the problems of training with few datasets and generalizing in a wild-environmental condition. Based on our analysis, we describe certain existing challenges in the use of FSL in FER systems and suggest research directions to resolve these issues. FER using FSL can be time efficient and reduce the complexity in many other real-time processing tasks and is an important area for further research.

Vision-Autocorrect: A Self-Adapting Approach towards Relieving Eye-Strain Using Facial-Expression Recognition

The last two years have seen a rapid rise in the duration of time that both adults and children spend on screens, driven by the recent COVID-19 health pandemic. A key adverse effect is digital eye strain (DES). Recent trends in human-computer interaction and user experience have proposed voice or gesture-guided designs that present more effective and less intrusive automated solutions. These approaches inspired the design of a solution that uses facial expression recognition (FER) techniques to detect DES and autonomously adapt the application to enhance the user’s experience. This study sourced and adapted popular open FER datasets for DES studies, trained convolutional neural network models for DES expression recognition, and designed a self-adaptive solution as a proof of concept. Initial experimental results yielded a model with an accuracy of 77% and resulted in the adaptation of the user application based on the FER classification results. We also provide the developed application, model source code, and adapted dataset used for further improvements in the area. Future work should focus on detecting posture, ergonomics, or distance from the screen.

Intelligent facial expression recognition and classification using optimal deep transfer learning model

Facial expression is commonly utilized by humans to deliver their mood and emotional state to other people. Facial expression recognition (FER) becomes a hot research area in recent days, and it is a tedious process owing to the presence of high intra-class variation. The conventional methods for FEC are mainly based on handcrafted features with a classification model trained on image or video datasets. Since the facial datasets involve large variations in the images and comprise partial faces, it is needed to design automated FER models. The latest advancements in artificial intelligence (AI) and deep learning (DL) models find useful for better understanding of facial emotions related to face images. In this aspect, this paper presents an intelligent FER using optimal deep transfer learning (IFER-DTFL) model. The proposed IFER-DTFL technique aims to detect the face and identify the facial expressions automatically. The IFER-DTFL technique encompasses a three state process: face detection, feature extraction, and expression classification. In addition, a mask RCNN model is used for the detection of faces. Moreover, the Adam optimizer with Densely Connected Networks (DenseNet121) model is employed for feature extraction process. Furthermore, the weighted kernel extreme learning machine (WKELM) model is utilized to classify the facial expressions. A comprehensive set of simulations were carried out on benchmark dataset and the results are inspected under varying aspects. The experimental results pointed out the supremacy of the IFER-DTFL technique over the other recent techniques interms of several performance measures. • Propose a deep learning based facial expression recognition technique. • Apply Mask RCNN model for face detection process. • Employ Adam optimizer with DenseNet and WKELM model for recognition.

A comprehensive review of facial expression recognition techniques

A comprehensive review of facial expression recognition techniques

A Bird\'s Eye View of Human Facial Expression Recognition Techniques

bstract: Human Face Expression Recognition is one of the most remarkable and promising errands in friendly interaction. For the most part, facial emotions are normal and straight forward means for human creatures to impart their expressions and goals. Facial emotions are the critical attributes of non-verbal interactions. In this paper depicts the review of Face Expression Recognition (FER) strategies which incorporate the three significant steps, for example, preprocessing, face detection and emotion classification. In this review work, clarifies the different kinds of FER procedures with its significant handout. Till now many works is done in Facial expression recognition. This paper will in general concentrate on the work done in recognition utilizing various methods. Facial recognition is significant viewpoint in making more human like PCs where human intelligent machine can be construct. Various methods are utilized till now on various viewpoints; in this paper we will be mainly focusing on feature based facial expression recognition. This paper additionally centers on classification strategies.

Impact of Activation, Optimization, and Regularization Methods on the Facial Expression Model Using CNN.

When it comes to conveying sentiments and thoughts, facial expressions are quite effective. For human-computer collaboration, data-driven animation, and communication between humans and robots to be successful, the capacity to recognize emotional states in facial expressions must be developed and implemented. Recently published studies have found that deep learning is becoming increasingly popular in the field of image categorization. As a result, to resolve the problem of facial expression recognition (FER) using convolutional neural networks (CNN), increasingly substantial efforts have been made in recent years. Facial expressions may be acquired from databases like CK+ and JAFFE using this novel FER technique based on activations, optimizations, and regularization parameters. The model recognized emotions such as happiness, sadness, surprise, fear, anger, disgust, and neutrality. The performance of the model was evaluated using a variety of methodologies, including activation, optimization, and regularization, as well as other hyperparameters, as detailed in this study. In experiments, the FER technique may be used to recognize emotions with an Adam, Softmax, and Dropout Ratio of 0.1 to 0.2 when combined with other techniques. It also outperforms current FER techniques that rely on handcrafted features and only one channel, as well as has superior network performance compared to the present state-of-the-art techniques.

Measuring the Candidates' Emotions in Political Debates Based on Facial Expression Recognition Techniques.

This article presents the analysis of the main Spanish political candidates for the elections to be held on April 2019. The analysis focuses on the Facial Expression Analysis (FEA), a technique widely used in neuromarketing research. It allows to identify the micro-expressions that are very brief, involuntary. They are signals of hidden emotions that cannot be controlled voluntarily. The video with the final interventions of every candidate has been post-processed using the classification algorithms given by the iMotions's AFFDEX platform. We have then analyzed these data. Firstly, we have identified and compare the basic emotions showed by each politician. Second, we have associated the basic emotions with specific moments of the candidate's speech, identifying the topics they address and relating them directly to the expressed emotion. Third, we have analyzed whether the differences shown by each candidate in every emotion are statistically significant. In this sense, we have applied the non-parametric chi-squared goodness-of-fit test. We have also considered the ANOVA analysis in order to test whether, on average, there are differences between the candidates. Finally, we have checked if there is consistency between the results provided by different surveys from the main media in Spain regarding the evaluation of the debate and those obtained in our empirical analysis. A predominance of negative emotions has been observed. Some inconsistencies were found between the emotion expressed in the facial expression and the verbal content of the message. Also, evidences got from statistical analysis confirm that the differences observed between the various candidates with respect to the basic emotions, on average, are statistically significant. In this sense, this article provides a methodological contribution to the analysis of the public figures' communication, which could help politicians to improve the effectiveness of their messages identifying and evaluating the intensity of the expressed emotions.