Facial Expression Recognition Model Research Articles

Open-Set Facial Expression Recognition

Facial expression recognition (FER) models are typically trained on datasets with a fixed number of seven basic classes. However, recent research works (Cowen et al. 2021; Bryant et al. 2022; Kollias 2023) point out that there are far more expressions than the basic ones. Thus, when these models are deployed in the real world, they may encounter unknown classes, such as compound expressions that cannot be classified into existing basic classes. To address this issue, we propose the open-set FER task for the first time. Though there are many existing open-set recognition methods, we argue that they do not work well for open-set FER because FER data are all human faces with very small inter-class distances, which makes the open-set samples very similar to close-set samples. In this paper, we are the first to transform the disadvantage of small inter-class distance into an advantage by proposing a new way for open-set FER. Specifically, we find that small inter-class distance allows for sparsely distributed pseudo labels of open-set samples, which can be viewed as symmetric noisy labels. Based on this novel observation, we convert the open-set FER to a noisy label detection problem. We further propose a novel method that incorporates attention map consistency and cycle training to detect the open-set samples. Extensive experiments on various FER datasets demonstrate that our method clearly outperforms state-of-the-art open-set recognition methods by large margins. Code is available at https://github.com/zyh-uaiaaaa.

Towards the adversarial robustness of facial expression recognition: Facial attention-aware adversarial training

Beyond the in-the-lab environment, deep-learning-based facial expression recognition (FER) models that provide reliable performance on wild datasets are gradually becoming applied to the real world. However, the fact that neural networks are inherently vulnerable to digital attacks (e.g., adversarial examples) and their performance is not exposed to external threats reduces the applicability of FER technology. So, we design a so-called test-time attack scenario in which FER models are deceived by superimposing imperceptible perturbation(s) on test images. This scenario, which targets the testing phase in which model weakness is revealed, clearly shows how vulnerable FER models are to external attacks. As a remedy against this attack, we propose a novel method called FAAT, which adversarially trains the model by paying attention to core region(s) of face. FAAT aims to improve model robustness so that the model can be generalized to unseen perturbation(s) while focusing on facial expression-related areas. For example, FAAT’s robustness against PGD attack with a performance improvement of up to 18% is encouraging. Also, various benchmarking results based on our attack scenario analyze the fidelity of prior arts and will promote the development direction of future models.

Exploration of Facial Emotion Detection Systems Utilizing Convolutional Neural Networks: A Comprehensive Review

Facial emotion detection systems have witnessed significant advancements, particularly with the utilization of convolutional neural networks (CNNs). This paper provides a thorough survey of such systems, beginning with an introduction to artificial intelligence and the evolutionary trajectory of neural networks, including artificial neural networks (ANNs), recurrent neural networks (RNNs), and CNNs. The paper elaborates on CNNs' architecture and functionality, elucidating key components such as convolutional layers, pooling layers, and fully connected layers, while also spotlighting prominent CNN architectures like AlexNet and ResNet. It delineates the broad scope and diverse applications of facial emotion detection systems across various domains, including marketing research, crowd testing, AI robots, banking, and entertainment. In the literature review section, recent research papers on CNN models for facial expression recognition are synthesized, highlighting variances in datasets, methodologies, and accuracy levels. The paper concludes that CNNs represent the current pinnacle of facial emotion classification techniques, surpassing previous methodologies such as eigenfaces. It underscores the efficacy of deep CNN architectures trained on extensive facial image datasets in proficiently identifying emotions from facial expressions. Moreover, the paper emphasizes the necessity for ongoing endeavors to enhance accuracy, particularly concerning complex emotions like disgust. In essence, CNNs exhibit substantial promise for the development of real-world facial emotion detection systems, heralding a new era of sophisticated emotion recognition technology.

Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and Datasets

Recent technological developments have enabled computers to identify and categorize facial expressions to determine a person’s emotional state in an image or a video. This process, called “Facial Expression Recognition (FER)”, has become one of the most popular research areas in computer vision. In recent times, deep FER systems have primarily concentrated on addressing two significant challenges: the problem of overfitting due to limited training data availability, and the presence of expression-unrelated variations, including illumination, head pose, image resolution, and identity bias. In this paper, a comprehensive survey is provided on deep FER, encompassing algorithms and datasets that offer insights into these intrinsic problems. Initially, this paper presents a detailed timeline showcasing the evolution of methods and datasets in deep facial expression recognition (FER). This timeline illustrates the progression and development of the techniques and data resources used in FER. Then, a comprehensive review of FER methods is introduced, including the basic principles of FER (components such as preprocessing, feature extraction and classification, and methods, etc.) from the pro-deep learning era (traditional methods using handcrafted features, i.e., SVM and HOG, etc.) to the deep learning era. Moreover, a brief introduction is provided related to the benchmark datasets (there are two categories: controlled environments (lab) and uncontrolled environments (in the wild)) used to evaluate different FER methods and a comparison of different FER models. Existing deep neural networks and related training strategies designed for FER, based on static images and dynamic image sequences, are discussed. The remaining challenges and corresponding opportunities in FER and the future directions for designing robust deep FER systems are also pinpointed.

BPMB: BayesCNNs with perturbed multi-branch structure for robust facial expression recognition

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets.

3DEmo: For Portrait Emotion Recognition with New Dataset

Emotional Expression Recognition (EER) and Facial Expression Recognition (FER) are active research areas in the affective computing field, which involves studying human emotion, recognition, and sentiment analysis. The main objective of this research is to develop algorithms that can accurately interpret and estimate human emotions from portrait images. The emotions depicted in a portrait can reflect various factors such as psychological and physiological states, the artist’s emotional responses, social and environmental aspects, and the period in which the painting was created. This task is challenging because (i) the portraits are often depicted in an artistic or stylized manner rather than realistically or naturally, (ii) the texture and color features obtained from natural faces and paintings differ, affecting the success rate of emotion recognition algorithms, and (iii) it is a new research area, where practically we do not have visual arts portrait facial emotion estimation models or datasets. To address these challenges, we need a new class of tools and a database specifically tailored to analyze portrait images. This study aims to develop art portrait emotion recognition methods and create a new digital portrait dataset containing 927 images. The proposed model is based on (i) a 3-dimensional estimation of emotions learned by a deep neural network and (ii) a novel deep learning module (3DEmo) that could be easily integrated into existing FER models. To evaluate the effectiveness of the developed models, we also tested their robustness on a facial emotion recognition dataset. The extensive simulation results show that the presented approach outperforms established methods. We expect that this dataset and the developed new tools will encourage further research in recognizing emotions in portrait paintings and predicting artists’ emotions in the painting period based on their artwork.

Deep learning strategies with CReToNeXt-YOLOv5 for advanced pig face emotion detection.

This study underscores the paramount importance of facial expressions in pigs, serving as a sophisticated mode of communication to gauge their emotions, physical well-being, and intentions. Given the inherent challenges in deciphering such expressions due to pigs' rudimentary facial muscle structure, we introduced an avant-garde pig facial expression recognition model named CReToNeXt-YOLOv5. The proposed model encompasses several refinements tailored for heightened accuracy and adeptness in detection. Primarily, the transition from the CIOU to the EIOU loss function optimized the training dynamics, leading to precision-driven regression outcomes. Furthermore, the incorporation of the Coordinate Attention mechanism accentuated the model's sensitivity to intricate expression features. A significant innovation was the integration of the CReToNeXt module, fortifying the model's prowess in discerning nuanced expressions. Efficacy trials revealed that CReToNeXt-YOLOv5 clinched a mean average precision (mAP) of 89.4%, marking a substantial enhancement by 6.7% relative to the foundational YOLOv5. Crucially, this advancement holds profound implications for animal welfare monitoring and research, as our findings underscore the model's capacity to revolutionize the accuracy of pig facial expression recognition, paving the way for more humane and informed livestock management practices.

A Transfer Learning Approach for Facial Emotion Recognition Using a Deep Learning Model

The facial expression recognition (FER) system is the process of identifying the emotional state of a person. Emotion recognition from facial expressions is a rapidly growing area of research with numerous applications, including psychology, marketing, and human-computer interaction. This paper presents a novel approach to FER employing deep learning techniques, specifically leveraging the power of transfer learning methodology. To address the challenge of accurately recognizing different emotions, including angry, disgusted, afraid, happy, sad, surprised, and neutral, from facial expressions, a pre-trained model MobileNetV2 architecture has been fine-tuned for improving accuracy. It is a lightweight convolutional neural network architecture, specifically designed for efficient on-device inference. For evaluating the performance of the proposed FER model, the state-of-the-art FER-2013 dataset, along random images and video clips has been employed. Experimental results demonstrate that the proposed FER model attained a remarkable accuracy rate exceeding 99% when tested on diverse sets of random images and video clips. Moreover, the system achieved a notable accuracy of 61% when evaluated against the FER-2013 dataset. Overall, this approach presents a significant advancement in the field of real-time facial expression recognition using the MobileNetV2 architecture with FER2013 dataset, which may improve the quality of human-computer interactions.

Adaptive graph-based feature normalization for facial expression recognition

Facial Expression Recognition (FER) suffers from data uncertainties caused by ambiguous facial images and annotators’ subjectiveness, resulting in excursive semantic and feature covariate shifting problems. Existing works usually correct mislabeled data by estimating noise distribution, or guide network training with knowledge domain that learned from clean data, neglecting the associative relations of expression samples. In this work, we propose an Adaptive Graph-based Feature Normalization (AGFN) to protect FER models from data uncertainties by normalizing feature distributions with the association of expressions. Specifically, we propose a Poisson graph generator to adaptively construct topological graphs for samples in each mini-batches via a sampling process, and correspondingly design a coordinate descent strategy to optimize proposed model. Our method outperforms state-of-the-art works with accuracies of 91.84%, 91.11% and 61.38% on three benchmark datasets, i.e., FERPlus, RAF-DB and AffectNet. Especially, when the percentage of mislabeled data significantly increases (e.g., to 20%), our method surpasses existing works by 14.09%, 21.12% and 13.67% on above datasets. Our code is publicly available in https://github.com/X-Lab-CN/AGBFN.

FaceTopoNet: Facial Expression Recognition Using Face Topology Learning

Prior work has shown that the order in which different components of the face are learned using a sequential learner can play an important role in the performance of facial expression recognition systems. We propose FaceTopoNet, an end- to-end deep model for facial expression recognition, which is capable of learning an effective tree topology of the face. Our model then traverses the learned tree to generate a sequence, which is then used to form an embedding to feed a sequential learner. The devised model adopts one stream for learning structure and one stream for learning texture. The structure stream focuses on the positions of the facial landmarks, while the main focus of the texture stream is on the patches around the landmarks to learn textural information. We then fuse the outputs of the two streams by utilizing an effective attention-based fusion strategy. We perform extensive experiments on four large- scale in-the-wild facial expression datasets - namely AffectNet, FER2013, ExpW, and RAF-DB - and one lab-controlled dataset (CK+) to evaluate our approach. FaceTopoNet achieves state-of- the-art performance on three of the five datasets and obtains competitive results on the other two datasets. We also perform rigorous ablation and sensitivity experiments to evaluate the impact of different components and parameters in our model. Lastly, we perform robustness experiments and demonstrate that FaceTopoNet is more robust against occlusions in comparison to other leading methods in the area.

Classroom Facial Expression Recognition Method Based on Conv3D-ConvLSTM-SEnet in Online Education Environment

A Facial Expression Recognition (FER) method based on Conv3D-ConvLSTM-SEnet in an online education environment is proposed to address the issue of low accuracy in current classroom expression recognition (FER) methods. Firstly, ConvLSTM is used to integrate the local feature extraction ability of CNN and the temporal modeling ability of LSTM, and based on this, the rich spatial features of the image are characterized. Then, by introducing Depth Separated Convolution (DSC) to change the number of output channels and separate each channel, the Feature Maps (F-M) are sequentially concatenated to obtain a multi-channel output F-M. Finally, based on ConvLSTM and SEnet modules, a Conv3D ConvLSTM-based FER model was proposed by redistributing the extracted abstract features and basic texture features, achieving high accuracy classroom FER. The proposed classroom FER method and the other five methods were compared and analyzed through simulation experiments using the CK[Formula: see text], FER2013 and JAFFE datasets, respectively. The results indicate that the proposed method achieves the highest accuracy, precision, recall, and F1 score, with improvements of at least 1.85%, 2.41%, 1.18% and 2.05%, respectively, compared to the other five methods on the FER2013 dataset. The proposed method can perform FER on students in online classroom teaching, detect their emotions in real-time, and help teachers make better course teaching adjustments based on this.

Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognition

Facial expression recognition (FER) is one of the most important computer vision tasks for understanding human inner emotions. However, the poor generation ability of the FER model limits its applicability due to tremendous intraclass variation. Especially for expressions of varying intensities, the appearance differences among weak expressions are subtle, which makes FER tasks challenging. In response to these issues, this paper presents a dual subspace manifold learning method based on a graph convolutional network (GCN) for intensity-invariant FER tasks. Our method treats the target task as a node classification problem and learns the manifold representation using two subspace analysis methods: locality preserving projection (LPP) and peak-piloted locality preserving projection (PLPP). Inspired by the classic LPP, which maintains local similarity among data, this paper introduces a novel PLPP that maintains the locality between peak expressions and non-peak expressions to enhance the representation of weak expressions. This paper also reports two subspace fusion methods, one based on a weighted adjacency matrix and another on a self-attention mechanism, that combine the LPP and PLPP to further improve FER performance. The second method achieves a recognition accuracy of 93.83% on the CK+, 74.86% on the Oulu-CASIA and 75.37% on the MMI for weak expressions, outperforming state-of-the-art methods.

Cross-Domain Facial Expression Recognition through Reliable Global–Local Representation Learning and Dynamic Label Weighting

Cross-Domain Facial Expression Recognition (CD-FER) aims to develop a facial expression recognition model that can be trained in one domain and deliver consistent performance in another. CD-FER poses a significant challenges due to changes in marginal and class distributions between source and target domains. Existing methods primarily emphasize achieving domain-invariant features through global feature adaptation, often neglecting the potential benefits of transferable local features across different domains. To address this issue, we propose a novel framework for CD-FER that combines reliable global–local representation learning and dynamic label weighting. Our framework incorporates two key modules: the Pseudo-Complementary Label Generation (PCLG) module, which leverages pseudo-labels and complementary labels obtained using a credibility threshold to learn domain-invariant global and local features, and the Label Dynamic Weight Matching (LDWM) module, which assesses the learning difficulty of each category and adaptively assigns corresponding label weights, thereby enhancing the classification performance in the target domain. We evaluate our approach through extensive experiments and analyses on multiple public datasets, including RAF-DB, FER2013, CK+, JAFFE, SFW2.0, and ExpW. The experimental results demonstrate that our proposed model outperforms state-of-the-art methods, with an average accuracy improvement of 3.5% across the five datasets.

Effect of the deep residual networks at different depths on expression recognition

Facial expression recognition plays a critical role in numerous applications like emotion analysis, human-computer interaction, and surveillance systems. Given the importance of this task, this study aims to investigate the effectiveness of different depths of Residual Networks (ResNet). The primary objective is to scrutinize and compare these ResNet models in terms of their training and validation losses and performance metrics like accuracy, recall, and F1 scores. In this research, a thorough comparative analysis is conducted by setting up exhaustive experiments using these models. The experiment is carried out on a popular facial expression dataset. Despite the depth differences, ResNet101 emerged as the model demonstrating superior performance. It struck the most effective balance between model complexity and generalization capacity, leading to the lowest validation loss and better performance. Experimental results show that a more complex model does not necessarily yield optimal results. The optimal balance between model complexity and generalisation needs to be investigated. These findings can provide essential guidance in the design of deep learning models for facial expression recognition and other similar tasks.

ASPP-YOLOv5: A study on constructing pig facial expression recognition for heat stress

In the global pig farming industry, the decrease in growth rate of fattening pigs during hot seasons is a common and challenging issue. Especially in hot and humid environments, pigs are prone to heat stress, which can even lead to death in severe cases, causing significant economic losses to the farming industry. Therefore, timely detection of heat stress and taking appropriate measures is of great importance for saving the pigs. The ASPP-YOLOv5 model for pig heat stress facial expression recognition enhances the feature extraction capability by incorporating coordinated attention mechanisms. The CReToNeXt module is used to improve feature extraction and fusion, thereby enhancing the model's representational capacity and performance. The ASPP module better adapts to the multi-scale features of pig heat stress expressions, improves the network's perception of different-scale expression features, and enhances the model's recognition ability for pig heat stress expressions. The mean Average Precision(mAP) reached 93.2 %, representing a 7 % improvement compared to the YOLOv5s model. It also outperformed the Faster R-CNN and YOLOv4 models by achieving a remarkable 47.7 % and 48.8 % increase, respectively, in mAP. This model can be applied to a pig farm's heat stress unmanned and contactless real-time monitoring system, enabling timely warning for heat stress.

Facial Emotion Recognition and Classification Using the Convolutional Neural Network-10 (CNN-10)

The importance of facial expressions in nonverbal communication is significant because they help better represent the inner emotions of individuals. Emotions can depict the state of health and internal wellbeing of individuals. Facial expression detection has been a hot research topic in the last couple of years. The motivation for applying the convolutional neural network-10 (CNN-10) model for facial expression recognition stems from its ability to detect spatial features, manage translation invariance, understand expressive feature representations, gather global context, and achieve scalability, adaptability, and interoperability with transfer learning methods. This model offers a powerful instrument for reliably detecting and comprehending facial expressions, supporting usage in recognition of emotions, interaction between humans and computers, cognitive computing, and other areas. Earlier studies have developed different deep learning architectures to offer solutions to the challenge of facial expression recognition. Many of these studies have good performance on datasets of images taken under controlled conditions, but they fall short on more difficult datasets with more image diversity and incomplete faces. This paper applied CNN-10 and ViT models for facial emotion classification. The performance of the proposed models was compared with that of VGG19 and INCEPTIONV3. The CNN-10 outperformed the other models on the CK + dataset with a 99.9% accuracy score, FER-2013 with an accuracy of 84.3%, and JAFFE with an accuracy of 95.4%.

An Adaptive Fuzzy Ensemble Model for Facial Expression Recognition Using Poplar Optimization and CRNN

Facial expression recognition (FER) is one of the trending topics in the research field in recent years. Facial expressions are the direct way in which humans communicate their emotions and identifying these emotions requires complex cognitive skills. Because of continual advances in image processing and machine learning techniques, several novel strategies are presented each year. Their performance still needs to be enhanced, and it is a challenging task. So, we aim to propose a novel method called Ensemble Convolutional Recurrent Neural Network (CRNN), and this model is proposed to enhance the prediction accuracy of facial expressions. The methods such as GRU, LSTM, and CNN prediction outputs are incorporated. An Adaptive Neuro-Fuzzy Inference System (ANFIS) is constructed as the ensemble model's topmost layer to adequately examine the few sub-models. The final output is predicted by the fuzzy inference system. The facial Recognition Dataset and EMOTIC dataset are utilized in the method for evaluating the results. The various performance metrics are utilized to test the efficiency of the proposed model such as precision, False Positive Rate, F1-score, True Positive Rate, and accuracy. Our proposed method achieves 99.52% for precision and 99.35% for F1-score. Compared with other existing such our proposed method achieves 99.75% and the value of the Area under the ROC curve (AUC) is 0.95.

An Efficient Model for Facial Expression Recognition with Music Recommendation

An Efficient Model for Facial Expression Recognition with Music Recommendation

An extensive study of facial expression recognition using artificial intelligence techniques with different datasets

<p>Machine and deep learning (DL) algorithms have advanced to a point where a wide range of crucial real-world computer vision problems can be solved. Facial Expression Recognition (FER) is one of these applications; it is the foremost non-verbal intentions and a fascinating study of symmetry. A prevalent application of deep learning has become the area of vision, where facial expression recognition has emerged as one of the most promising new frontiers. Latterly deep learning-based FER models have been plagued by technical problems, including under-fitting and over-fitting. Probably inadequate information is used for training and expressing ideas. With these considerations in mind, this article gives a systematic and complete survey of the most cutting-edge AI strategies and gives a conclusion to address the aforementioned problems. It is also a scheme of classification for existing facial proposals in compact. This survey analyses the structure of the usual FER method and discusses the feasible technologies that may be used in its respective elements. In addition, this study provides a summary of seventeen widely-used FER datasets that reviews functioning novel machine and DL networks suggested by academics and outline their benefits and liability in the context of facial expression acknowledgment based on static replicas. Finally, this study discusses the research obstacles and open consequences of that well-conditioned face expression recognition scheme.</p>

Analysis and Recognition of Voluntary Facial Expression Mimicry Based on Depressed Patients.

Many clinical studies have shown that facial expression recognition and cognitive function are impaired in depressed patients. Different from spontaneous facial expression mimicry (SFEM), 164 subjects (82 in a case group and 82 in a control group) participated in our voluntary facial expression mimicry (VFEM) experiment using expressions of neutrality, anger, disgust, fear, happiness, sadness and surprise. Our research is as follows. First, we collected a large amount of subject data for VFEM. Second, we extracted the geometric features of subject facial expression images for VFEM and used Spearman correlation analysis, a random forest, and logistic regression-based recursive feature elimination (LR-RFE) to perform feature selection. The features selected revealed the difference between the case group and the control group. Third, we combined geometric features with the original images and improved the advanced deep learning facial expression recognition (FER) algorithms in different systems. We propose the E-ViT and E-ResNet based on VFEM. The accuracies and F1 scores were higher than those of the baseline models, respectively. Our research proved that it is effective to use feature selection to screen geometric features and combine them with a deep learning model for depression facial expression recognition.