Facial Expression Recognition Model Research Articles

Triple attention feature enhanced pyramid network for facial expression recognition

The purpose of facial expression recognition is to capture facial expression features from static pictures or videos and to provide the most intuitive information about human emotion changes for artificial intelligence devices to use effectively for human-computer interaction. Among the factors, the excessive loss of locally valid information and the irreversible degradation trend of the information at different expression semantic scales with increasing network depth are the main challenges faced currently. To address such problems, an enhanced pyramidal network model combining with triple attention mechanisms is designed in this paper. Firstly, three attention mechanism modules, i.e. CBAM, SK, and SE, are embedded into the backbone network model in stages, and the key features are sensed by using spatial or channel information mining, which effectively reduces the effective information loss caused by the network depth. Then, the pyramid network is used as an extension of the backbone network to obtain the semantic information of expression features across scales. The recognition accuracy reaches 96.25% and 73.61% in the CK+ and Fer2013 expression change datasets, respectively. Furthermore, by comparing with other current advanced methods, it is shown that the proposed network architecture combining with the triple attention mechanism and multi-scale cross-information fusion can simultaneously maintain and improve the information mining ability and recognition accuracy of the facial expression recognition model.

Students’ Facial Expression Recognition Based on Multi-head Attention Mechanism

As there are a far greater number of students in the classroom than teachers, it is difficult for teachers to grasp the learning of all students. Especially during the epidemic period, when online teaching becomes a trend, this problem is more prominent. The issue of how to grasp the learning of each student more comprehensively remains a problem to be solved in teaching. Under such circumstances, students’ facial expression recognition emerges as one of the most important solutions. In this paper, an improved facial expression recognition model based on the multi-head attention mechanism is proposed. The model is tested on two student expression databases, the JAFFE and the OL-SFED, and the recognition rate reaches 99.5% and 100%, respectively. In addition, to compare it with the models developed by other researchers, it is also tested on the RAF-DB, recording the best recognition rate of 90.35% and an average recognition rate of 83.66%, which represents the best level so far.

Face Expression Recognition using Deep Neural Network

The research topic concerned in this paper is creating a machine learning model for facial expression recognition (FER). It is a technology to which uses biometric markers to detect emotions in human faces. FER is important because of human-computer interaction, pattern recognition and image recognition. Three components are containing in our method, including dataset construction, model building and Emoji generation. First, the authors aim to build a hand-crafted Convolutional Neural Network to recognize the emotion and designed a GUI in which generates the corresponding emoji derived from the facial expression put in. This paper tested our model in the popular FER2013 dataset and our model achieves 89% accuracy on the dataset, which validates the satisfying performance of our method. Our method can both achieve satisfying accuracy on the dataset and show its strong application ability in most of the tasks. The method can be applied to the real-world scenes for expression recognition and emoji generation.

Online facial expression recognition based on graph convolution and long short memory networks

Video‐based facial expression recognition (FER) models have achieved higher accuracy with more computation, which is not suitable for online deployment in mobile intelligent terminals. Facial landmarks can model facial expression changes with their spatial location information instead of texture features. But classical convolution operation cannot make full use of landmark information. To this end, in this paper, we propose a novel long short memory network (LSTM) by embedding graph convolution named GELSTM for online video‐based FER in mobile intelligent terminals. Specifically, we construct landmark‐based face graph data from the client. On the server side, we introduce graph convolution which can effectively mine spatial dependencies information in a landmark‐based facial graph. Moreover, the extracted landmark's features are fed to LSTM for temporal feature aggregation. We conduct experiments on the facial expression dataset and the results show our proposed method shows superior performance compared to other deep models.

Deep learning based MobileNet and multi-head attention model for facial expression recognition

Facial expressions is an intuitive reflection of a person’s emotional state, and it is one of the most important forms of interpersonal communication. Due to the complexity and variability of human facial expressions, traditional methods based on handcrafted feature extraction have shown insufficient performances. For this purpose, we proposed a new system of facial expression recognition based on MobileNet model with the addition of skip connections to prevent the degradation in performance in deeper architectures. Moreover, multi-head attention mechanism was applied to concentrate the processing on the most relevant parts of the image. The experiments were conducted on FER2013 database, which is imbalanced and includes ambiguities in some images containing synthetic faces. We applied a pre-processing step of face detection to eliminate wrong images, and we implemented both SMOTE and Near-Miss algorithms to get a balanced dataset and prevent the model to being biased. The experimental results showed the effectiveness of the proposed framework which achieved the recognition rate of 96.02% when applying multi-head attention mechanism

Multi-Net Fusion: Exploring a Brain-Inspired Neural Network Model for Facial Expression Recognition

Multi-Net Fusion: Exploring a Brain-Inspired Neural Network Model for Facial Expression Recognition

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

The learning status of learners directly affects the quality of learning. Compared with offline teachers, it is difficult for online teachers to capture the learning status of students in the whole class, and it is even more difficult to continue to pay attention to students while teaching. Therefore, this paper proposes an online learning state analysis model based on a convolutional neural network and multi-dimensional information fusion. Specifically, a facial expression recognition model and an eye state recognition model are constructed to detect students’ emotions and fatigue, respectively. By integrating the detected data with the homework test score data after online learning, an analysis model of students’ online learning status is constructed. According to the PAD model, the learning state is expressed as three dimensions of students’ understanding, engagement and interest, and then analyzed from multiple perspectives. Finally, the proposed model is applied to actual teaching, and procedural analysis of 5 different types of online classroom learners is carried out, and the validity of the model is verified by comparing with the results of the manual analysis.

Earthworm Optimization with Improved SqueezeNet Enabled Facial Expression Recognition Model

Facial expression recognition (FER) remains a hot research area among computer vision researchers and still becomes a challenge because of high intra-class variations. Conventional techniques for this problem depend on hand-crafted features, namely, LBP, SIFT, and HOG, along with that a classifier trained on a database of videos or images. Many execute perform well on image datasets captured in a controlled condition; however not perform well in the more challenging dataset, which has partial faces and image variation. Recently, many studies presented an endwise structure for facial expression recognition by utilizing DL methods. Therefore, this study develops an earthworm optimization with an improved SqueezeNet-based FER (EWOISN-FER) model. The presented EWOISN-FER model primarily applies the contrast-limited adaptive histogram equalization (CLAHE) technique as a pre-processing step. In addition, the improved SqueezeNet model is exploited to derive an optimal set of feature vectors, and the hyperparameter tuning process is performed by the stochastic gradient boosting (SGB) model. Finally, EWO with sparse autoencoder (SAE) is employed for the FER process, and the EWO algorithm appropriately chooses the SAE parameters. A wide-ranging experimental analysis is carried out to examine the performance of the proposed model. The experimental outcomes indicate the supremacy of the presented EWOISN-FER technique.

Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills

In this paper, the proposed work tests the computer vision application to perform the skill and emotion assessment of children with Autism Spectrum Disorder (ASD) by extracting various bio-behaviors, human activities, child-therapist interactions, and joint pose estimations from the video-recorded interactive single-or two-person play-based intervention sessions. A comprehensive data set of 300 videos are amassed from ASD children engaged in social interaction and developed three novel deep learning-based computer vision models which are explained as follows: 1) activity comprehension to analyze child-play partner interactions (Activity Comprehension model); 2) an automatic joint attention recognition framework using pose, and 3) emotion and facial expression recognition. We tested models on children’s real-world unseen 68 videos captured from the clinic and public datasets. The activity comprehension model has an overall accuracy of 72.32%, the joint attention models have an accuracy of 97% for following eye gaze and 93.4% for hand pointing and the facial expression recognition model has an overall accuracy of 95.1%. The proposed models could extract activities and behaviors of interest from free-play and intervention session videos, empowering clinicians with data useful in diagnosis, assessment, treatment formulation, and monitoring of ASD children with limited supervision.

Recognition of human emotions in real time

This article emphasizes the importance and relevance of using neural networks to recognize human emotions in real time. He explores the possibilities of using convolutional neural networks for efficient analysis and classification of emotions based on facial expressions. The article describes the process of developing an application using JavaScript and TensorFlow and highlights potential optimization opportunities to improve the speed and accuracy of emotion recognition. The solution proposed in this paper offers a highly effective approach to emotion recognition, applicable in a variety of fields ranging from medicine and social research to marketing and entertainment. This highlights the importance of advanced technologies that improve human-computer interaction and reflects the importance of using innovative methods to improve the quality of human life. The convolutional neural network (CNN) algorithm used in this study uses a deep learning approach for facial expression recognition. The architecture of the CNN model includes several convolutional layers, each of which is responsible for identifying specific features of face images, such as edges, textures, and patterns. These layers are followed by pooling layers to reduce dimensionality, allowing the network to focus on the most meaningful features. In addition, the use of rectified linear units (ReLU) as activation functions improves the model’s ability to capture complex relationships in the data. In addition, the paper examines a deep learning facial expression recognition (FER) model based on convolutional neural networks, demonstrating its effectiveness in accurately identifying and classifying different emotional states based on facial features. The robust architecture of the FER model and the learning process allow it to distinguish the nuances of facial expressions, contributing to a more detailed understanding of human emotions. Consequently, such a system could be useful for medicine, social research, marketing, and entertainment, helping to analyze emotional responses and improve user interaction strategies.

Knowledge Conditioned Variational Learning for One-Class Facial Expression Recognition.

The openness of application scenarios and the difficulties of data collection make it impossible to prepare all kinds of expressions for training. Hence, detecting expression absent during the training (called alien expression) is important to enhance the robustness of the recognition system. So in this paper, we propose a facial expression recognition (FER) model, named OneExpressNet, to quantify the probability that a test expression sample belongs to the distribution of training data. The proposed model is based on variational auto-encoder and enjoys several merits. First, different from conventional one class classification protocol, OneExpressNet transfers the useful knowledge from the related domain as a constraint condition of the target distribution. By doing so, OneExpressNet will pay more attention to the descriptive region for FER. Second, features from both source and target tasks will aggregate after constructing a skip connection between the encoder and decoder. Finally, to further separate alien expression from training expression, empirical compact variation loss is jointly optimized, so that training expression will concentrate on the compact manifold of feature space. The experimental results show that our method can achieve state-of-the-art results in one class facial expression recognition on small-scale lab-controlled datasets including CFEE and KDEF, and large-scale in-the-wild datasets including RAF-DB and ExpW.

Facial Expression Recognition Model Depending on Optimized Support Vector Machine

Facial Expression Recognition Model Depending on Optimized Support Vector Machine

A Facial Expression Recognition Model using Lightweight Dense-Connectivity Neural Networks for Monitoring Online Learning Activities

A Facial Expression Recognition Model using Lightweight Dense-Connectivity Neural Networks for Monitoring Online Learning Activities

The Efficient-CapsNet model for facial expression recognition

The Efficient-CapsNet model for facial expression recognition

Intelligent facial expression recognition and classification using optimal deep transfer learning model

Facial expression is commonly utilized by humans to deliver their mood and emotional state to other people. Facial expression recognition (FER) becomes a hot research area in recent days, and it is a tedious process owing to the presence of high intra-class variation. The conventional methods for FEC are mainly based on handcrafted features with a classification model trained on image or video datasets. Since the facial datasets involve large variations in the images and comprise partial faces, it is needed to design automated FER models. The latest advancements in artificial intelligence (AI) and deep learning (DL) models find useful for better understanding of facial emotions related to face images. In this aspect, this paper presents an intelligent FER using optimal deep transfer learning (IFER-DTFL) model. The proposed IFER-DTFL technique aims to detect the face and identify the facial expressions automatically. The IFER-DTFL technique encompasses a three state process: face detection, feature extraction, and expression classification. In addition, a mask RCNN model is used for the detection of faces. Moreover, the Adam optimizer with Densely Connected Networks (DenseNet121) model is employed for feature extraction process. Furthermore, the weighted kernel extreme learning machine (WKELM) model is utilized to classify the facial expressions. A comprehensive set of simulations were carried out on benchmark dataset and the results are inspected under varying aspects. The experimental results pointed out the supremacy of the IFER-DTFL technique over the other recent techniques interms of several performance measures. • Propose a deep learning based facial expression recognition technique. • Apply Mask RCNN model for face detection process. • Employ Adam optimizer with DenseNet and WKELM model for recognition.

Multi-Feature Integrated Concurrent Neural Network for Human Facial Expression Recognition

<p>Facial expression helps to communicate between the people for conveying abundant information about human emotions. Facial expression classification is applied in various fields such as remote learning education, medical care, and smart traffic. However, due to the complexity and diversity of the facial emotions, the present facial expression recognition model causes a low recognition rate and it is hard to extract the precise features that are related to facial expression changes. In order to overcome this problem, we proposed Multi-feature Integrated Concurrent Neural Network (MICNN) which is significantly different from the single neural network architectures. It aggregates the prominent features of facial expressions by integrating the three kinds of networks such as Sequential Convolutional Neural Network (SCNN), Residual Dense Network (RDN), and Attention Residual Learning Network (ARLN) to enhance the accuracy rate of facial emotions detection system. Additionally, Local Binary Pattern (LBP) and Principal Component Analysis (PCA) are applied for representing the facial features and these features are combined with the texture features identified by the Gray-level Co-occurrence Matrix (GLCM). Finally, the integrated features are fed into softmax layer to classify the facial images. The experiments are carried out on benchmark datasets by applying k-fold cross-validation and the results demonstrate the superiority of the proposed model.</p> <p> </p>

Disentangling Identity and Pose for Facial Expression Recognition

Facial expression recognition (FER) is a challenging problem because the expression component is always entangled with other irrelevant factors, such as identity and head pose. In this work, we propose an identity and pose disentangled facial expression recognition (IPD-FER) model to learn more discriminative feature representation. We regard the holistic facial representation as the combination of identity, pose and expression. These three components are encoded with different encoders. For identity encoder, a well pre-trained face recognition model is utilized and fixed during training, which alleviates the restriction on specific expression training data in previous works and makes the disentanglement practicable on in-the-wild datasets. At the same time, the pose and expression encoder are optimized with corresponding labels. Combining identity and pose feature, a neutral face of input individual should be generated by the decoder. When expression feature is added, the input image should be reconstructed. By comparing the difference between synthesized neutral and expressional images of the same individual, the expression component is further disentangled from identity and pose. Experimental results verify the effectiveness of our method on both lab-controlled and in-the-wild databases and we achieve state-of-the-art recognition performance.

EFFICIENTNET FOR HUMAN FER USING TRANSFER LEARNING

Automatic facial expression recognition (FER) remained a challenging problem in computer vision. Recognition of human facial expression is difficult for machine learning techniques since there is a variation in emotional expression from person to person. With the advancement in deep learning and the easy availability of digital data, this process has become more accessible. We proposed an efficient facial expression recognition model based EfficientNet as backbone architecture and trained the proposed model using the transfer learning technique. In this work, we have trained the network on publicly available emotion datasets (RAF-DB, FER-2013, CK+). We also used two ways to compare our trained model: inner and cross-data comparisons. In an internal comparison, the model achieved an accuracy of 81.68 % on DFEW and 71.02 % on FER-2013. In a cross-data comparison, the model trained on RAF-DB and tested on CK+ achieved 78.59%, while the model trained on RAF-DB and tested on FER-2013 achieved 56.10% accuracy. Finally, we generated an t-SEN distribution of our model on both datasets to demonstrate the model''s inter-class discriminatory power.

Emotional Intelligence in Humanoid Robots: Social and Technical Applications

Main challenge in the human–robot interaction is to endow the robots with emotional intelligence making the interaction more genuine and natural. Main aspect in achieving this goal is the robot’s capability to interpret human emotions. Destiny humanoid robot will be one of the most widely used agents in human-robot interaction. Destiny robot will be equipped with a module of facial expression recognition based on CNN (convolutional neural network) facial expression recognition model with high accuracy and capability. This module will be integrated into Destiny's SDK that will allow facial expression classification. Another approach is developing a tool for emotion recognition through vocal features, such as tone, timbre, pitch or loudness making it possible to acquire machine intelligence with emotional intelligence. This feature would be especially helpful for humans suffering from emotional disorders or from the inability to express emotions. The model can detect emotional signals in the human voice using CNN.

Facial expression recognition on partially occluded faces using component based ensemble stacked CNN.

Facial Expression Recognition (FER) is the basis for many applications including human-computer interaction and surveillance. While developing such applications, it is imperative to understand human emotions for better interaction with machines. Among many FER models developed so far, Ensemble Stacked Convolution Neural Networks (ES-CNN) showed an empirical impact in improving the performance of FER on static images. However, the existing ES-CNN based FER models trained with features extracted from the entire face, are unable to address the issues of ambient parameters such as pose, illumination, occlusions. To mitigate the problem of reduced performance of ES-CNN on partially occluded faces, a Component based ES-CNN (CES-CNN) is proposed. CES-CNN applies ES-CNN on action units of individual face components such as eyes, eyebrows, nose, cheek, mouth, and glabella as one subnet of the network. Max-Voting based ensemble classifier is used to ensemble the decisions of the subnets in order to obtain the optimized recognition accuracy. The proposed CES-CNN is validated by conducting experiments on benchmark datasets and the performance is compared with the state-of-the-art models. It is observed from the experimental results that the proposed model has a significant enhancement in the recognition accuracy compared to the existing models.