Facial Expression Recognition Research Research Articles

Performance analysis of deep unified model for facial expression recognition using convolution neural network

Facial expression recognition has gathered substantial attention in computer vision applications, with the need for robust and accurate models that can decipher human emotions from facial images. Performance analysis of a novel hybrid model combines the strengths of residual network (ResNet) and dense network (DenseNet) architectures after applying preprocessing for facial expression recognition. The proposed hybrid model capitalizes on the complementary characteristics of ResNet's and DenseNet's densely connected blocks to enhance the model's capacity to extract discriminative features from facial images. This research evaluates the hybrid model performance and conducts a comprehensive benchmark against established facial expression recognition convolution neural network (CNN) models. The analysis encompasses key aspects of model performance, including classification accuracy, and adaptability with the LFW dataset for facial expressions such as Anger, Fear, Happy, Disgust, Sad, Surprise, along Neutral. The research observes that the proposed hybrid model is more accurate and efficient computationally than existing models consistently. This performance analysis eliminates information on the hybrid model's perspective to further facial expression recognition research.

A comparative study of deep convolutional neural network-based facial expression regression

Facial expression is an important way for people to express their emotions. Recently, with the advancement of the computer vision, facial expression recognition has become a current research hotspot and has made significant progress, which can be utilized in the interaction between humans and computers, emotional computing and other computer vision fields, the evolution of artificial intelligence and deep learning has better promoted the research of facial expression recognition. Conventional approaches are largely based on machine learning, which leverages artificial way for feature extraction. The extracted facial expression features are interfered by human factors, so that the trained classifier cannot effectively interpret the expression information, which ultimately leads to insufficient model generalization ability and low recognition accuracy. However, deep learning-based facial expression recognition in real scenes is still affected by factors such as human pose, different degrees of facial occlusion, background environment and light interference, and the recognition accuracy still needs to be further improved. This paper focuses on representative convolutional neural network architectures and summarizes the strengths, weaknesses, and innovations of these networks, through which the advancement of neural network architectures shows the great potential of the direction.

Facial Expression Recognition Based on Deep Learning: A Review

This review paper provides a comprehensive analysis of recent advancements in Facial Expression Recognition (FER) through various deep learning models. Seven state-of-the-art models are scrutinized, each offering unique contributions to the field. The MBCC-CNN model demonstrates improved recognition rates on diverse datasets, addressing the challenges of facial expression recognition through multiple branches and cross-connected convolutional neural networks. The Deep Graph Fusion model introduces a novel approach for predicting viewer expressions from videos, showcasing superior performance on the EEV database. Multimodal emotion recognition is explored in the EEG and facial expression fusion model, achieving high accuracy on the DEAP dataset. The Spark-based LDSP-TOP descriptor, coupled with a 1-D CNN and LSTM Autoencoder, excels in capturing temporal dynamics for facial expression understanding. Vision transformers for micro-expression recognition exhibit outstanding accuracy on datasets like CASMEI, CASME-II, and SAMM. Additionally, a hierarchical deep learning model is proposed for evaluating teaching states based on facial expressions. Lastly, a visionary transformer model achieves remarkable recognition accuracy of 100% on SAMM dataset, showcasing the potential of combining convolutional and transformer architectures. This review synthesizes key findings, highlights model performances, and outlines directions for future research in FER.

Optimization of facial expression recognition based on dual attention mechanism by lightweight network model

As a deep learning network model, ResNet50 can effectively recognize facial expressions to a certain extent, but there are still problems such as insufficient extraction of local effective feature information and a large number of parameters. In this paper, we take ResNet50 as the basic framework to optimize and improve this network. Firstly, by analyzing the influence mechanism of the attention mechanism module on the network feature information circulation, the optimal embedding position of CBAM (Convolutional Block Attention Module) and SE modules in the ResNet50 network is thus determined to effectively extract local key information, and then the number of model parameters is effectively reduced by embedding the depth separable module. To validate the performance of the improved ResNet50 model, the recognition accuracy reached 71.72% and 95.72% by ablation experiments using Fer2013 and CK+ datasets, respectively. We then used the trained model to classify the homemade dataset, and the recognition accuracy reached 92.86%. In addition, compared with the current more advanced methods, the improved ResNet50 network model proposed in this paper can maintain a balance between model complexity and recognition ability and can provide a technical reference for facial expression recognition research.

Emotional Classification Based on Facial Expression Recognition Using Convolutional Neural Network Method

In recent years, the development of human-computer interaction technology has reached remarkable levels, particularly in the field of facial expression recognition. This technology utilizes human facial images to identify and classify emotional expressions such as happiness, sadness, fear, and more through computer image processing. Active research in facial expression recognition yields substantial benefits for individual and societal advancement, especially in the context of its application within Smart City environments. This study demonstrates that well- configured Convolutional Neural Network (CNN) models empowered by TensorFlow exhibit higher accuracy compared to models utilizing PyTorch. The TensorFlow model achieves the highest accuracy of 93% in recognizing emotional expressions, whereas the PyTorch model achieves 69% accuracy. The TensorFlow model also displays lower accuracy loss and shorter training times compared to the PyTorch model. In the context of calculating happiness indices within Smart City environments, the appropriate choice of technology significantly influences measurement accuracy and efficiency. Therefore, the TensorFlow platform, proven to deliver superior performance in this study, can be a strategic choice for integrating facial expression detection technology into happiness index measurements in such locations

A Survey of Deep Learning-based Facial Expression Recognition Research

Facial expression is one of the ways to convey emotional expression. Deep learning is used to analyze facial expression to understand people's true feelings, and human-computer interaction is integrated into it. However, in the natural real environment and various interference (such as lighting, age and ethnicity), facial expression recognition will face many challenges. In recent years, with the development of artificial intelligence, scholars have studied more and more facial expression recognition in the case of interference, which not only promotes the theoretical research, but also makes it popularized in the application. Facial expression recognition is to identify facial expressions to carry out emotion analysis, and emotion analysis can be analyzed with the help of facial expressions, speech, text, video and other signals. Therefore, facial expression recognition can be regarded as a research direction of emotion analysis. This paper focuses on the perspective of facial expression recognition to summarize. In the process of facial expression recognition, researchers usually try to combine multiple modal information such as voice, text, picture and video for analysis. Due to the differences between single-modal data set and multi-modal data set, this paper will analyze static facial expression recognition, dynamic facial expression recognition and multi-modal fusion. This research has a wide range of applications, such as: smart elderly care, medical research, detection of fatigue driving and other fields.

Lightweight Facial Expression Recognition Based on Class-Rebalancing Fusion Cumulative Learning

In the research of Facial Expression Recognition (FER), the inter-class of facial expression data is not evenly distributed, the features extracted by networks are insufficient, and the FER accuracy and speed are relatively low for practical applications. Therefore, a lightweight and efficient method based on class-rebalancing fusion cumulative learning for FER is proposed in our research. A dual-branch network (Regular feature learning and Rebalancing-Cumulative learning Network, RLR-CNet) is proposed, where the RLR-CNet uses the improvement in the lightweight ShuffleNet with two branches (feature learning and class-rebalancing) based on cumulative learning, which improves the efficiency of our model recognition. Then, to enhance the generalizability of our model and pursue better recognition efficiency in real scenes, a random masking method is improved to process datasets. Finally, in order to extract local detailed features and further improve FER efficiency, a shuffle attention module (SA) is embedded in the model. The results demonstrate that the recognition accuracy of our RLR-CNet is 71.14%, 98.04%, and 87.93% on FER2013, CK+, and RAF-DB, respectively. Compared with other FER methods, our method has great recognition accuracy, and the number of parameters is only 1.02 MB, which is 17.74% lower than that in the original ShuffleNet.

Multimodal sentiment system and method based on CRNN-SVM

Traditional sentiment analysis focuses on text-level sentiment mining, transforming sentiment mining into classification or regression problems, resulting in a sentiment analysis low accuracy rate. Sentiment analysis refers to the use of natural language processing, text analysis, and computational linguistics to systematically identify, extract, quantify, and study sentimental states. Therefore, more scholars have begun to focus on speech recognition and facial expression recognition research, and extracting and analysing people’s sentiment tendencies can improve sentiment recognition accuracy. Traditional single-modal sentiment analysis can no longer meet people’s needs. Therefore, this paper proposes a multimodal sentiment analysis method based on the multimodal sentiment analysis method that can obtain more sentimental information sources and help people make better decisions. The experimental results in this paper show that the highest recognition rates of CNN-SVM, RNN-SVM, and CRNN-SVM were 76.8%, 71.2%, and 93.5%, respectively. It can be seen that CRNN-SVM has the highest sentiment tendency recognition rate in deep learning, so it is suitable to apply CRNN-SVM to sentiment tendency analysis system design in this paper. The average accuracy rate of the system designed in this paper was 91%, and the stability was also very strong, which shows that the system designed in this paper is meaningful. The main contribution of this paper is based on the limitations of single-mode emotion analysis. It proposes a multimode emotion analysis method and introduces a convolutional neural network to help people obtain more emotional information sources to meet their needs.

Expression Recognition Based on Multi-level Multi-model Fusion Deep Convolutional Neural Network

In the research of Facial Expression Recognition (FER), training networks using deep learning methods often relies on data sets with accurate label and balanced number of each classification. However, when training on low-quality data sets, the performance of traditional models is insufficient. In order to solve the problem, the paper proposes an expression recognition method based on Multi-level Multi-model Fusion Deep Convolutional Neural Network. The Deep Convolutional Neural Network is composed of three models, ResNet-50, ResNet-101 and InceptionNet-V3, which have similar performance on ImageNet. We use the weighted average fusion algorithm as our fusion method. According to the accuracy of the three models, weighted fusion is carried out, and the model with high accuracy is given higher weight. It increases the width of the network and the number of parameters. At the same time, it can solve the problem of accuracy degradation when the number of layers in the deep network increases. We choose FER2013, a widely recognized low-quality data set, for the experiment. The experimental results based on FER2013 data set show that the facial expression recognition accuracy of this method reach 71.78%, which is significantly better than ResNet-50, ResNet-101 and InceptionNet-V3. Compared with other models, it also verifies the effectiveness of the proposed model in identifying low-quality data sets.

Facial expression recognition using visual transformer with histogram of oriented gradients

Emotions play an important role in our life as a response to our interactions with others, decisions, and so on. Among various emotional signals, facial expression is one of the most powerful and natural means for humans to convey their emotions and intentions, and it has the advantage of easily obtaining information using only a camera, so facial expression-based emotional research is being actively conducted. Facial expression recognition(FER) have been studied by classifying them into seven basic emotions: anger, disgust, fear, happiness, sadness, surprise, and normal. Before the appearance of deep learning, handcrafted feature extractors and simple classifiers such as SVM, Adaboost was used to extracted Facial emotion. With the advent of deep learning, it is now possible to extract facial expression without using feature extractors. Despite its excellent performance in FER research, it is still challenging task due to external factors such as occlusion, illumination, and pose, and similarity problems between different facial expressions. In this paper, we propose a method of training through a ResNet [1] and Visual Transformer [2] called FViT and using Histogram of Oriented Gradients(HOGs) [3] data to solve the similarity problem between facial expressions.

Facial Expression Recognition Using Dual Path Feature Fusion and Stacked Attention

Facial Expression Recognition (FER) can achieve an understanding of the emotional changes of a specific target group. The relatively small dataset related to facial expression recognition and the lack of a high accuracy of expression recognition are both a challenge for researchers. In recent years, with the rapid development of computer technology, especially the great progress of deep learning, more and more convolutional neural networks have been developed for FER research. Most of the convolutional neural performances are not good enough when dealing with the problems of overfitting from too-small datasets and noise, due to expression-independent intra-class differences. In this paper, we propose a Dual Path Stacked Attention Network (DPSAN) to better cope with the above challenges. Firstly, the features of key regions in faces are extracted using segmentation, and irrelevant regions are ignored, which effectively suppresses intra-class differences. Secondly, by providing the global image and segmented local image regions as training data for the integrated dual path model, the overfitting problem of the deep network due to a lack of data can be effectively mitigated. Finally, this paper also designs a stacked attention module to weight the fused feature maps according to the importance of each part for expression recognition. For the cropping scheme, this paper chooses to adopt a cropping method based on the fixed four regions of the face image, to segment out the key image regions and to ignore the irrelevant regions, so as to improve the efficiency of the algorithm computation. The experimental results on the public datasets, CK+ and FERPLUS, demonstrate the effectiveness of DPSAN, and its accuracy reaches the level of current state-of-the-art methods on both CK+ and FERPLUS, with 93.2% and 87.63% accuracy on the CK+ dataset and FERPLUS dataset, respectively.

Facial expression recognition via coarse-grained and fine-grained feature representation

Recognizing facial expressions rely on facial parts’ movement (action units) such as eyes, mouth, and nose. Existing methods utilize complex subnetworks to learn part-based facial features or train neural networks with an extensively perturbed dataset. Different from existing methods, we propose a trainable end-to-end convolutional neural network for facial expression recognition. First, we propose a Local Prediction Penalty to stimulate facial expression recognition research with no part-based learning. It is a technique to punish the feature extractor’s local predictive power to coerce it to learn coarse-grained features (general facial expression). The Local Prediction Penalty forces the network to disregard predictive local signals learned from local receptive fields and instead depend on the global facial region. Second, we propose a Spatial Self-Attention method for fine-grained feature representation to learn distinct face features from pixel positions. The Spatial Self-Attention accumulates attention features at privileged positions without changing the spatial feature dimension. Lastly, we leverage a classifier to carefully combine all learned features (coarse-grained and fine-grained) for better feature representation. Extensive experiments demonstrate that our proposed methods significantly improve facial expression recognition performance.

Facial Expression Recognition from a Single Face Image Based on Deep Learning and Broad Learning

With advances in computer vision and artificial intelligence technology, facial expression recognition research has become a prominent topic. Current research is grappling with how to enable computers to fully understand expression features and improve recognition rates. Most single face image datasets are based on the psychological classification of the six basic human expressions used for network training. By outlining the problem of facial recognition by comparing traditional methods, deep learning, and broad learning techniques, this review highlights the remaining challenges and future directions of deep learning and broad learning research. The deep learning method has made it easier and more effective to extract expression features and improve facial expression recognition accuracy by end-to-end feature learning, but there are still many difficulties in robustness and real-time performance. The broad learning system (BLS) is a broad network structure that is expanded by increasing the number of feature nodes and enhancement nodes appropriately to reinforce the structure and is also effective in facial expression recognition. However, outliers and noises in unbalanced datasets need BLS to solve in the future. Finally, we present several problems that still need to be addressed in facial expression recognition.

Research on Expression Recognition Algorithm Based on Attention Mechanism

Facial expressions carry the emotional state of human beings and are one of the most powerful signals for humans to express themselves. Therefore, the research of facial expression recognition has very important significance and broad application prospects. However, in complex scenes, the blurring of facial images, the interference of redundant information, and the similarity of expressions will cause unsatisfactory expression recognition. In response to these shortcomings, this article adds a convolutional attention module to the basic Resnet network structure, infers the attention map in turn along two independent dimensions (channel and space), performs adaptive feature optimization, captures expression feature information, and reduces human faces. The interference of redundant information; and the introduction of island loss ISlandloss classification to optimize the distance between classes, so that the network can learn more discriminative features, and improve the discrimination of expressions. Compared with the existing mainstream facial expression classification algorithms, this method has certain advantages in objective evaluation indicators. The experimental results on FER2013 show that the accuracy of this algorithm is 73.85%, which is higher than 71.23% of the Resnet network.

Research on Facial Expression Recognition Algorithm Based on CNN

In order to realize the recognition of facial expression quickly and accurately, the traditional convolutional neural network is improved in this paper, and the research of facial expression recognition based on LeNet-5 convolutional neural network algorithm is proposed. Firstly, the image is preprocessed, including normalization and geometric transformation. Secondly, the convolution neural network model based on LeNet-5 model is designed, in which the Flatten layer is added, the Dropout strategy is introduced, and the LeakReLU is used as the activation function. At last, the experimental results show that the recognition rate of the improved algorithm is better than that of LBP, sift and hog, but it takes the longest time and the largest amount of computation.

Efficient facial expression recognition based on convolutional neural network

The goal of research in Facial Expression Recognition (FER) is to build a robust and strong recognizability model. In this paper, we propose a new scheme for FER systems based on convolutional neural network. Part of the regular convolution operation is replaced by depthwise separable convolution to reduce the number of parameters and the computational workload; the self-adaption joint loss function is adopted to improve the classification performance. In addition, we balance our train set through data augmentation, and we preprocess the input images through illumination processing, face detection, and other methods, effectively maximizing the expression recognition rate. Experiments to validate our methods are conducted based on the TensorFlow platform and Fer2013 dataset. We analyze the experimental results before and after train set balancing and network model modification, and we compare our results with those of other researchers. The results show that our method is effective at increasing the expression recognition rate under the same experiment conditions. We further conduct an experiment on our own expression dataset relevant to driving safety, and it yields similar results.

FERCE: Facial Expression Recognition for Combined Emotions Using FERCE Algorithm

Research in facial expression recognition and emotion detection has undergone many developments recently. Human beings express different feelings via facial expressions at different circumstances. Perceiving such type of communication helps us in building a computational human cognition model. There are seven basic emotions: anger, disgust, fear, happy, sad, surprise and neutral. Researchers have stuck to those seven emotions for research into facial expression recognition and emotion detection. Apart from those basic emotions, the existing literature has defined 21 other compound emotions. However, they failed to explore some more emotions and few have been eliminated by claiming them as meaningless. This study proposes an algorithm that offers a new approach for grouping the emotion classes. In this proposed work a total of 37 emotions are identified as combined emotions out of which 16 are newly derived. These derived emotions were unnoticed by the earlier and they are existing in real-time human emotions. Along with exploring new emotions, the claim that certain existing emotions are symmetric is challenged and proved to be asymmetric in nature. This claim is validated by employing the Facial Action Coding System (FACS) and statistical analysis of combined emotions.

Research and development of autism diagnosis information system based on deep convolution neural network and facial expression data

PurposeFacial expression provides abundant information for social interaction, and the analysis and utilization of facial expression data are playing a huge driving role in all areas of society. Facial expression data can reflect people's mental state. In health care, the analysis and processing of facial expression data can promote the improvement of people's health. This paper introduces several important public facial expression databases and describes the process of facial expression recognition. The standard facial expression database FER2013 and CK+ were used as the main training samples. At the same time, the facial expression image data of 16 Chinese children were collected as supplementary samples. With the help of VGG19 and Resnet18 algorithm models of deep convolution neural network, this paper studies and develops an information system for the diagnosis of autism by facial expression data.Design/methodology/approachThe facial expression data of the training samples are based on the standard expression database FER2013 and CK+. FER2013 and CK+ databases are a common facial expression data set, which is suitable for the research of facial expression recognition. On the basis of FER2013 and CK+ facial expression database, this paper uses the machine learning model support vector machine (SVM) and deep convolution neural network model CNN, VGG19 and Resnet18 to complete the facial expression recognition.FindingsIn this study, ten normal children and ten autistic patients were recruited to test the accuracy of the information system and the diagnostic effect of autism. After testing, the accuracy rate of facial expression recognition is 81.4 percent. This information system can easily identify autistic children. The feasibility of recognizing autism through facial expression is verified.Research limitations/implicationsThe CK+ facial expression database contains some adult facial expression images. In order to improve the accuracy of facial expression recognition for children, more facial expression data of children will be collected as training samples. Therefore, the recognition rate of the information system will be further improved.Originality/valueThis research uses facial expression data and the latest artificial intelligence technology, which is advanced in technology. The diagnostic accuracy of autism is higher than that of traditional systems, so this study is innovative. Research topics come from the actual needs of doctors, and the contents and methods of research have been discussed with doctors many times. The system can diagnose autism as early as possible, promote the early treatment and rehabilitation of patients, and then reduce the economic and mental burden of patients. Therefore, this information system has good social benefits and application value.

Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition.

Occlusion and pose variations, which can change facial appearance significantly, are two major obstacles for automatic Facial Expression Recognition (FER). Though automatic FER has made substantial progresses in the past few decades, occlusion-robust and pose-invariant issues of FER have received relatively less attention, especially in real-world scenarios. This paper addresses the real-world pose and occlusion robust FER problem in the following aspects. First, to stimulate the research of FER under real-world occlusions and variant poses, we annotate several in-the-wild FER datasets with pose and occlusion attributes for the community. Second, we propose a novel Region Attention Network (RAN), to adaptively capture the importance of facial regions for occlusion and pose variant FER. The RAN aggregates and embeds varied number of region features produced by a backbone convolutional neural network into a compact fixed-length representation. Last, inspired by the fact that facial expressions are mainly defined by facial action units, we propose a region biased loss to encourage high attention weights for the most important regions. We validate our RAN and region biased loss on both our built test datasets and four popular datasets: FERPlus, AffectNet, RAF-DB, and SFEW. Extensive experiments show that our RAN and region biased loss largely improve the performance of FER with occlusion and variant pose. Our method also achieves state-of-the-art results on FERPlus, AffectNet, RAF-DB, and SFEW. Code and the collected test data will be publicly available.

Three convolutional neural network models for facial expression recognition in the wild

Facial expression recognition (FER) in the wild is a novel and challenging topic in the field of human emotion perception. Different kinds of convolutional neural network (CNN) approaches have been applied to this topic, but few of them ever considered what kind of architecture was better for the FER research. In this paper, we proposed three novel CNN models with different architectures. The first one is a shallow network, named the Light-CNN, which is a fully convolutional neural network consisting of six depthwise separable residual convolution modules to solve the problem of complex topology and over-fitting. The second one is a dual-branch CNN which extracts traditional LBP features and deep learning features in parallel. The third one is a pre-trained CNN which is designed by transfer learning technique to overcome the shortage of training samples. Extensive evaluations on three popular datasets (public CK+, multi-view BU-3DEF and FER2013 datasets) demonstrated that our models were competitive and representative in the field of FER in the wild research. We achieved significant better results with comparisons to plenty of state-of-the-art approaches. Moreover, we provided discussions on the effectiveness and practicability of CNNs with different feature types and architectures for FER in the wild as well.