Facial Expression Recognition Model Research Articles

Adaptive graph-based feature normalization for facial expression recognition

Facial Expression Recognition (FER) suffers from data uncertainties caused by ambiguous facial images and annotators’ subjectiveness, resulting in excursive semantic and feature covariate shifting problems. Existing works usually correct mislabeled data by estimating noise distribution, or guide network training with knowledge domain that learned from clean data, neglecting the associative relations of expression samples. In this work, we propose an Adaptive Graph-based Feature Normalization (AGFN) to protect FER models from data uncertainties by normalizing feature distributions with the association of expressions. Specifically, we propose a Poisson graph generator to adaptively construct topological graphs for samples in each mini-batches via a sampling process, and correspondingly design a coordinate descent strategy to optimize proposed model. Our method outperforms state-of-the-art works with accuracies of 91.84%, 91.11% and 61.38% on three benchmark datasets, i.e., FERPlus, RAF-DB and AffectNet. Especially, when the percentage of mislabeled data significantly increases (e.g., to 20%), our method surpasses existing works by 14.09%, 21.12% and 13.67% on above datasets. Our code is publicly available in https://github.com/X-Lab-CN/AGBFN.

FaceTopoNet: Facial Expression Recognition Using Face Topology Learning

Prior work has shown that the order in which different components of the face are learned using a sequential learner can play an important role in the performance of facial expression recognition systems. We propose FaceTopoNet, an end- to-end deep model for facial expression recognition, which is capable of learning an effective tree topology of the face. Our model then traverses the learned tree to generate a sequence, which is then used to form an embedding to feed a sequential learner. The devised model adopts one stream for learning structure and one stream for learning texture. The structure stream focuses on the positions of the facial landmarks, while the main focus of the texture stream is on the patches around the landmarks to learn textural information. We then fuse the outputs of the two streams by utilizing an effective attention-based fusion strategy. We perform extensive experiments on four large- scale in-the-wild facial expression datasets - namely AffectNet, FER2013, ExpW, and RAF-DB - and one lab-controlled dataset (CK+) to evaluate our approach. FaceTopoNet achieves state-of- the-art performance on three of the five datasets and obtains competitive results on the other two datasets. We also perform rigorous ablation and sensitivity experiments to evaluate the impact of different components and parameters in our model. Lastly, we perform robustness experiments and demonstrate that FaceTopoNet is more robust against occlusions in comparison to other leading methods in the area.

Classroom Facial Expression Recognition Method Based on Conv3D-ConvLSTM-SEnet in Online Education Environment

A Facial Expression Recognition (FER) method based on Conv3D-ConvLSTM-SEnet in an online education environment is proposed to address the issue of low accuracy in current classroom expression recognition (FER) methods. Firstly, ConvLSTM is used to integrate the local feature extraction ability of CNN and the temporal modeling ability of LSTM, and based on this, the rich spatial features of the image are characterized. Then, by introducing Depth Separated Convolution (DSC) to change the number of output channels and separate each channel, the Feature Maps (F-M) are sequentially concatenated to obtain a multi-channel output F-M. Finally, based on ConvLSTM and SEnet modules, a Conv3D ConvLSTM-based FER model was proposed by redistributing the extracted abstract features and basic texture features, achieving high accuracy classroom FER. The proposed classroom FER method and the other five methods were compared and analyzed through simulation experiments using the CK[Formula: see text], FER2013 and JAFFE datasets, respectively. The results indicate that the proposed method achieves the highest accuracy, precision, recall, and F1 score, with improvements of at least 1.85%, 2.41%, 1.18% and 2.05%, respectively, compared to the other five methods on the FER2013 dataset. The proposed method can perform FER on students in online classroom teaching, detect their emotions in real-time, and help teachers make better course teaching adjustments based on this.

Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognition

Facial expression recognition (FER) is one of the most important computer vision tasks for understanding human inner emotions. However, the poor generation ability of the FER model limits its applicability due to tremendous intraclass variation. Especially for expressions of varying intensities, the appearance differences among weak expressions are subtle, which makes FER tasks challenging. In response to these issues, this paper presents a dual subspace manifold learning method based on a graph convolutional network (GCN) for intensity-invariant FER tasks. Our method treats the target task as a node classification problem and learns the manifold representation using two subspace analysis methods: locality preserving projection (LPP) and peak-piloted locality preserving projection (PLPP). Inspired by the classic LPP, which maintains local similarity among data, this paper introduces a novel PLPP that maintains the locality between peak expressions and non-peak expressions to enhance the representation of weak expressions. This paper also reports two subspace fusion methods, one based on a weighted adjacency matrix and another on a self-attention mechanism, that combine the LPP and PLPP to further improve FER performance. The second method achieves a recognition accuracy of 93.83% on the CK+, 74.86% on the Oulu-CASIA and 75.37% on the MMI for weak expressions, outperforming state-of-the-art methods.

Cross-Domain Facial Expression Recognition through Reliable Global–Local Representation Learning and Dynamic Label Weighting

Cross-Domain Facial Expression Recognition (CD-FER) aims to develop a facial expression recognition model that can be trained in one domain and deliver consistent performance in another. CD-FER poses a significant challenges due to changes in marginal and class distributions between source and target domains. Existing methods primarily emphasize achieving domain-invariant features through global feature adaptation, often neglecting the potential benefits of transferable local features across different domains. To address this issue, we propose a novel framework for CD-FER that combines reliable global–local representation learning and dynamic label weighting. Our framework incorporates two key modules: the Pseudo-Complementary Label Generation (PCLG) module, which leverages pseudo-labels and complementary labels obtained using a credibility threshold to learn domain-invariant global and local features, and the Label Dynamic Weight Matching (LDWM) module, which assesses the learning difficulty of each category and adaptively assigns corresponding label weights, thereby enhancing the classification performance in the target domain. We evaluate our approach through extensive experiments and analyses on multiple public datasets, including RAF-DB, FER2013, CK+, JAFFE, SFW2.0, and ExpW. The experimental results demonstrate that our proposed model outperforms state-of-the-art methods, with an average accuracy improvement of 3.5% across the five datasets.

Effect of the deep residual networks at different depths on expression recognition

Facial expression recognition plays a critical role in numerous applications like emotion analysis, human-computer interaction, and surveillance systems. Given the importance of this task, this study aims to investigate the effectiveness of different depths of Residual Networks (ResNet). The primary objective is to scrutinize and compare these ResNet models in terms of their training and validation losses and performance metrics like accuracy, recall, and F1 scores. In this research, a thorough comparative analysis is conducted by setting up exhaustive experiments using these models. The experiment is carried out on a popular facial expression dataset. Despite the depth differences, ResNet101 emerged as the model demonstrating superior performance. It struck the most effective balance between model complexity and generalization capacity, leading to the lowest validation loss and better performance. Experimental results show that a more complex model does not necessarily yield optimal results. The optimal balance between model complexity and generalisation needs to be investigated. These findings can provide essential guidance in the design of deep learning models for facial expression recognition and other similar tasks.

ASPP-YOLOv5: A study on constructing pig facial expression recognition for heat stress

In the global pig farming industry, the decrease in growth rate of fattening pigs during hot seasons is a common and challenging issue. Especially in hot and humid environments, pigs are prone to heat stress, which can even lead to death in severe cases, causing significant economic losses to the farming industry. Therefore, timely detection of heat stress and taking appropriate measures is of great importance for saving the pigs. The ASPP-YOLOv5 model for pig heat stress facial expression recognition enhances the feature extraction capability by incorporating coordinated attention mechanisms. The CReToNeXt module is used to improve feature extraction and fusion, thereby enhancing the model's representational capacity and performance. The ASPP module better adapts to the multi-scale features of pig heat stress expressions, improves the network's perception of different-scale expression features, and enhances the model's recognition ability for pig heat stress expressions. The mean Average Precision(mAP) reached 93.2 %, representing a 7 % improvement compared to the YOLOv5s model. It also outperformed the Faster R-CNN and YOLOv4 models by achieving a remarkable 47.7 % and 48.8 % increase, respectively, in mAP. This model can be applied to a pig farm's heat stress unmanned and contactless real-time monitoring system, enabling timely warning for heat stress.

Facial Emotion Recognition and Classification Using the Convolutional Neural Network-10 (CNN-10)

The importance of facial expressions in nonverbal communication is significant because they help better represent the inner emotions of individuals. Emotions can depict the state of health and internal wellbeing of individuals. Facial expression detection has been a hot research topic in the last couple of years. The motivation for applying the convolutional neural network-10 (CNN-10) model for facial expression recognition stems from its ability to detect spatial features, manage translation invariance, understand expressive feature representations, gather global context, and achieve scalability, adaptability, and interoperability with transfer learning methods. This model offers a powerful instrument for reliably detecting and comprehending facial expressions, supporting usage in recognition of emotions, interaction between humans and computers, cognitive computing, and other areas. Earlier studies have developed different deep learning architectures to offer solutions to the challenge of facial expression recognition. Many of these studies have good performance on datasets of images taken under controlled conditions, but they fall short on more difficult datasets with more image diversity and incomplete faces. This paper applied CNN-10 and ViT models for facial emotion classification. The performance of the proposed models was compared with that of VGG19 and INCEPTIONV3. The CNN-10 outperformed the other models on the CK + dataset with a 99.9% accuracy score, FER-2013 with an accuracy of 84.3%, and JAFFE with an accuracy of 95.4%.

An Adaptive Fuzzy Ensemble Model for Facial Expression Recognition Using Poplar Optimization and CRNN

Facial expression recognition (FER) is one of the trending topics in the research field in recent years. Facial expressions are the direct way in which humans communicate their emotions and identifying these emotions requires complex cognitive skills. Because of continual advances in image processing and machine learning techniques, several novel strategies are presented each year. Their performance still needs to be enhanced, and it is a challenging task. So, we aim to propose a novel method called Ensemble Convolutional Recurrent Neural Network (CRNN), and this model is proposed to enhance the prediction accuracy of facial expressions. The methods such as GRU, LSTM, and CNN prediction outputs are incorporated. An Adaptive Neuro-Fuzzy Inference System (ANFIS) is constructed as the ensemble model's topmost layer to adequately examine the few sub-models. The final output is predicted by the fuzzy inference system. The facial Recognition Dataset and EMOTIC dataset are utilized in the method for evaluating the results. The various performance metrics are utilized to test the efficiency of the proposed model such as precision, False Positive Rate, F1-score, True Positive Rate, and accuracy. Our proposed method achieves 99.52% for precision and 99.35% for F1-score. Compared with other existing such our proposed method achieves 99.75% and the value of the Area under the ROC curve (AUC) is 0.95.

An Efficient Model for Facial Expression Recognition with Music Recommendation

An Efficient Model for Facial Expression Recognition with Music Recommendation

An extensive study of facial expression recognition using artificial intelligence techniques with different datasets

<p>Machine and deep learning (DL) algorithms have advanced to a point where a wide range of crucial real-world computer vision problems can be solved. Facial Expression Recognition (FER) is one of these applications; it is the foremost non-verbal intentions and a fascinating study of symmetry. A prevalent application of deep learning has become the area of vision, where facial expression recognition has emerged as one of the most promising new frontiers. Latterly deep learning-based FER models have been plagued by technical problems, including under-fitting and over-fitting. Probably inadequate information is used for training and expressing ideas. With these considerations in mind, this article gives a systematic and complete survey of the most cutting-edge AI strategies and gives a conclusion to address the aforementioned problems. It is also a scheme of classification for existing facial proposals in compact. This survey analyses the structure of the usual FER method and discusses the feasible technologies that may be used in its respective elements. In addition, this study provides a summary of seventeen widely-used FER datasets that reviews functioning novel machine and DL networks suggested by academics and outline their benefits and liability in the context of facial expression acknowledgment based on static replicas. Finally, this study discusses the research obstacles and open consequences of that well-conditioned face expression recognition scheme.</p>

Analysis and Recognition of Voluntary Facial Expression Mimicry Based on Depressed Patients.

Many clinical studies have shown that facial expression recognition and cognitive function are impaired in depressed patients. Different from spontaneous facial expression mimicry (SFEM), 164 subjects (82 in a case group and 82 in a control group) participated in our voluntary facial expression mimicry (VFEM) experiment using expressions of neutrality, anger, disgust, fear, happiness, sadness and surprise. Our research is as follows. First, we collected a large amount of subject data for VFEM. Second, we extracted the geometric features of subject facial expression images for VFEM and used Spearman correlation analysis, a random forest, and logistic regression-based recursive feature elimination (LR-RFE) to perform feature selection. The features selected revealed the difference between the case group and the control group. Third, we combined geometric features with the original images and improved the advanced deep learning facial expression recognition (FER) algorithms in different systems. We propose the E-ViT and E-ResNet based on VFEM. The accuracies and F1 scores were higher than those of the baseline models, respectively. Our research proved that it is effective to use feature selection to screen geometric features and combine them with a deep learning model for depression facial expression recognition.

Facial Expression Recognition Model Based on CNN and Data Augmentation Method

Face expressions are essential in expressing human emotions, and it is accomplished by separating features and categorizing them. Facial expression recognition technology has been widely employed in human-computer interaction, telemedicine, mental health, and criminal investigation detection. In recent years, significant advances in deep learning have facilitated the development of facial expression recognition, making it increasingly accessible. Convolutional neural networks (CNN) are the foundation of the face expression recognition model presented in this article. In order to maximize the final accuracy, an abundance of sample photographs are required for training and optimization. As a result, the 35,886 facial expression photos from the FER2013 dataset, which contains all seven emotions, were used for both training and testing. The photos were scaled down to 4848 pixels during data preparation, and data augmentation was carried out. To get accurate face expression recognition results, various optimizers and parameters were chosen for the training network, which was a bespoke structure based on the VGG network design. The model constructed in this study achieved an accuracy of 73.16% during prediction.

Hybrid heuristic mechanism for occlusion aware facial expression recognition scheme using patch based adaptive CNN with attention mechanism

In computer vision, the recognition of expressions from partially occluded faces is one of the serious problems. By the prior recognizing techniques it can solve the issue by various assumptions. A benchmark-guided branch was proposed for detecting and eliminating the manipulated features from the occluded regions since the human visual system is proficient for eliminating the occlusion and the appropriate focal point was obtained on the non-occluded areas. In recent years deep learning has attained a great place in the recognition of facial reactions Still, the precision of facial expression is affected by the occlusion and large skew. In this research work, a deep structure-based occlusionaware facial expression recognition mechanism is introduced to provide superior recognition results. Firstly, the required image is taken from publically provided online sources and the gathered images are subjected to the face extraction method. The face extraction method is done via the ViolaJones method for the extraction of redundant patterns from the original images. Secondly, the extracted face features are given to the pattern recognition stage, where the Adaptive CNN with Attention Mechanism (ACNN-AM) is introduced. This mechanism automatically forms the occluded region of the face and the focal point was on the most discriminative un-occluded regions. Moreover, the hidden patterns in the Occlusion aware facial expressions are identified through the Hybrid Galactic Swarm Yellow Saddle Goatfish Optimization (HGSYSGO). Finally, the overall effectiveness of the developed occlusion aware facial expression recognition model is examined through the comparative analysis of different existing baseline recognition techniques.

FER-CHC: Facial expression recognition with cross-hierarchy contrast

Facial expression recognition (FER) tasks with convolutional neural networks (CNNs) have seen remarkable progress. However, these CNN-based approaches do not well capture detailed and crucial features that can distinguish different facial expressions from a global perspective. There is still much room for improvement in the performance of existing CNN-based models for FER. To address this, we propose a novel cross-hierarchy contrast (CHC) framework called FER-CHC for FER tasks. FER-CHC employs a contrastive learning mechanism to utilize these crucial features in improving the performance of CNN-based models for FER. Specifically, FER-CHC utilizes CHC to regularize the feature learning of the backbone network and enhance global representations of facial expressions. The CHC captures common and differential features from different facial expressions with a cross-hierarchy contrast mechanism. Furthermore, a fusion network globally integrates the features learned from both the backbone network and CHC to learn a more robust feature representation. We conducted comprehensive experiments on six popular datasets: CK+, FER2013, FER+, RAF-DB, AffectNet, and JAFFE. The results show that our proposed FER-CHC achieves state-of-the-art performances on these datasets. Additionally, an ablation study was conducted to demonstrate the effectiveness of the proposed components in FER-CHC.

GCANet: Geometry cues-aware facial expression recognition based on graph convolutional networks

Facial expression recognition (FER) task in the wild is challenging due to some uncertainties, such as the ambiguity of facial expressions, subjective annotations, and low-quality facial images. A novel model for FER in-the-wild datasets is proposed in this study to solve these uncertainties. The overview of the proposed method is as follows. First, the facial images are grouped into high and low uncertainties by the pre-trained network. The graph convolutional network (GCN) framework is then used for the facial images with low uncertainty to obtain geometry cues, including the relationship among action units (AUs) and the implicit connection between AUs and expressions, which help predict the probability of the underlying emotional label. The emotion label distribution is produced by combining the predicted latent label probability and the given label. For the facial images with high uncertainty, k-nearest neighbor graphs are built to determine the k facial images in the low uncertainty group with the highest similarity to the given facial image. The emotion label distribution of the given image is then replaced by fusing the emotion label distribution based on the distances between the given image and its adjacent images. Finally, the constructed emotion label distribution facilitates training in a straightforward manner using a convolutional neural network framework to identify facial expressions. Experimental results on RAF-DB, FERPlus, AffectNet, and SFEW2.0 datasets demonstrate that the proposed method achieved superior performance compared to state-of-the-art approaches.

A novel driver emotion recognition system based on deep ensemble classification

Driver emotion classification is an important topic that can raise awareness of driving habits because many drivers are overconfident and unaware of their bad driving habits. Drivers will acquire insight into their poor driving behaviors and be better able to avoid future accidents if their behavior is automatically identified. In this paper, we use different models such as convolutional neural networks, recurrent neural networks, and multi-layer perceptron classification models to construct an ensemble convolutional neural network-based enhanced driver facial expression recognition model. First, the faces of the drivers are discovered using the faster region-based convolutional neural network (R-CNN) model, which can recognize faces in real-time and offline video reliably and effectively. The feature-fusing technique is utilized to integrate the features extracted from three CNN models, and the fused features are then used to train the suggested ensemble classification model. To increase the accuracy and efficiency of face detection, a new convolutional neural network block (InceptionV3) replaces the improved Faster R-CNN feature-learning block. To evaluate the proposed face detection and driver facial expression recognition (DFER) datasets, we achieved an accuracy of 98.01%, 99.53%, 99.27%, 96.81%, and 99.90% on the JAFFE, CK+, FER-2013, AffectNet, and custom-developed datasets, respectively. The custom-developed dataset has been recorded as the best among all under the simulation environment.

Facial emotion recognition using deep learning detector and classifier

Numerous research works have been put forward over the years to advance the field of facial expression recognition which until today, is still considered a challenging task. The selection of image color space and the use of facial alignment as preprocessing steps may collectively pose a significant impact on the accuracy and computational cost of facial emotion recognition, which is crucial to optimize the speed-accuracy trade-off. This paper proposed a deep learning-based facial emotion recognition pipeline that can be used to predict the emotion of detected face regions in video sequences. Five well-known state-of-the-art convolutional neural network architectures are used for training the emotion classifier to identify the network architecture which gives the best speed-accuracy trade-off. Two distinct facial emotion training datasets are prepared to investigate the effect of image color space and facial alignment on the performance of facial emotion recognition. Experimental results show that training a facial expression recognition model with grayscale-aligned facial images is preferable as it offers better recognition rates with lower detection latency. The lightweight MobileNet_v1 is identified as the best-performing model with WM=0.75 and RM=160 as its hyper-parameters, achieving an overall accuracy of 86.42% on the testing video dataset.

Facial expression recognition of masked faces using deep learning

Facial expression recognition (FER) represents one of the most prevalent forms of interpersonal communication, which contains rich emotional information. But it became even more challenging during the times of COVID, where face masks became a mandatory protection measure, leading to the challenge of occluded lower-face during facial expression recognition. In this study, deep convolutional neural network (DCNN) represents the core of both our full-face FER system and our masked face FER model. The focus was on incorporating knowledge distillation in transfer learning between a teacher model, which is the full-face FER DCNN, and the student model, which is the masked face FER DCNN via the combination of both the loss from the teacher soft-labels vs the student soft labels and the loss from the dataset hard-labels vs the student hard-labels. The teacher-student architecture used FER2013 and a masked customized version of FER2013 as datasets to generate an accuracy of 69% and 61% respectively. Therefore, the study proves that the process of knowledge distillation may be used as a way for transfer learning and enhancing accuracy as a regular DCNN model (student only) would result in 46% accuracy compared to our approach (61% accuracy).

Facial expression recognition for a seven-class small and medium-sized dataset based on transfer learning CNNs

As one of the most common biological information for human to express their emotions, facial expression plays an important role in biological research, psychological analysis and even human-computer interaction in the computer field. It is very important to use the efficient computing and processing power of computers to realize automatic facial expression recognition. However, in the research process of this field, most people's technology and model achievements are based on large datasets, which lack the universality of small and medium-sized datasets. Therefore, this project provides an optimization model on a specific seven-class small and medium face image dataset and provides a possible technical optimization reference direction for facial expression recognition models on similar small and medium-sized datasets. During the experiment, the training performance of VGG16 and MobileNet is compared. A comparative experiment is set up to observe the effect of transfer learning mechanism on training results. The results show that transfer learning has a significant effect on the performance of the model, and the accuracy of the optimal test set is more than 90%. Regardless of whether transfer learning mechanism is used, the training performance of VGG16 model structure is better than that of MobileNet structure in the same dataset.