Performance Of Facial Expression Recognition Research Articles

A gradual self distillation network with adaptive channel attention for facial expression recognition

Facial expression recognition (FER) is widely applied in various real-world applications, such as security, human–computer interaction, and healthcare, leading to high demands for fast and accurate FER techniques. However, it remains a challenging task to design effective and efficient FER techniques that meet the requirements of real-world applications. In this paper, we propose a lightweight Gradual Self Distillation Network with adaptive channel attention (GSDNet) for accurate and efficient FER. Especially, we propose a novel gradual self distillation strategy which enables the network to learn from itself in a gradual and adaptive way. Specifically, the proposed GSDNet consists of a feature extraction backbone with multiple basic blocks. We plug an adaptive classifier after each basic block. Every two neighbor classifiers form “student & teacher” relationship for gradual knowledge distillation. In particular, the gradual self distillation strategy enables the transfer of key knowledge from deep to shallow layers gradually. Besides, an Adaptive Channel Attention Module (ACAM) is designed to enhance the representation capability of each block for adaptively capturing important features and achieve better FER performance. Extensive experiments on three real-world datasets show that the proposed method GSDNet outperforms the baselines, including state-of-the-art methods. Specifically, the accuracy of GSDNet on the RAF-DB, Affect-net, and FERPlus datasets is 90.91%, 66.11%, and 90.32%, separately. The code is available at https://github.com/Emy-cv/GSDNet.

Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition

Illumination variation has been a long-term challenge in real-world facial expression recognition (FER). Under uncontrolled or non-visible light conditions, near-infrared (NIR) can provide a simple and alternative solution to obtain high-quality images and supplement the geometric and texture details that are missing in the visible (VIS) domain. Due to the lack of large-scale NIR facial expression datasets, directly extending VIS FER methods to the NIR spectrum may be ineffective. Additionally, previous heterogeneous image synthesis methods are restricted by low controllability without prior task knowledge. To tackle these issues, we present the first approach, called for NIR-FER Stochastic Differential Equations (NFER-SDE), that transforms face expression appearance between heterogeneous modalities to the overfitting problem on small-scale NIR data. NFER-SDE can take the whole VIS source image as input and, together with domain-specific knowledge, guide the preservation of modality-invariant information in the high-frequency content of the image. Extensive experiments and ablation studies show that NFER-SDE significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition

With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents a more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt at deep NIR facial expression recognition and propose a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we construct a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

FG-AGR: Fine-Grained Associative Graph Representation for Facial Expression Recognition in the Wild

Facial expression recognition (FER) in the wild is challenging due to various unconstrained conditions, i.e., occlusions and head pose variations. Previous methods tend to improve the performance of facial expression recognition through resorting to holistic methods or coarse local-based methods, while ignoring the local fine-grained feature structure knowledge and the correlation between features. In this paper, we propose a Fine-Grained Association Graph Representation (FG-AGR) framework which can capture the local fine-grained facial expression representation. Firstly, an Adaptive Salient Region Induction (ASRI) is designed for adaptively highlighting the local saliency regions of facial expressions combined with spatial location information. Based on this, a Local Fine-grained Feature Extraction (LFFE) based on Visual Transformers is introduced to further extract fine but discriminative fine-grained features of saliency regions. Thirdly, an Adaptive Graph Association Reasoning (AGAR) based on Graph Convolutional Network is constructed to learn associated fine-grained feature combinations. Extensive experiments demonstrate that our FG-AGR achieves superior performance compared to the state-of-the-art methods with 90.81% on RAF-DB, 64.91% on AffectNet-7, 60.69% on AffectNet-8 and 91.09% on FERPlus.

A joint learning method with consistency-aware for low-resolution facial expression recognition

Existing facial expression recognition (FER) methods are mainly devoted to learning discriminative features from high-resolution images. However, when applied to low-resolution images, their performance drops rapidly. This paper proposes a unified learning framework (namely SR-FER) by cascading the image super-resolution (SR) task and FER task to alleviate the low-resolution challenge. It effectively feeds back expression-related information from the FER network to the SR network, and returns the quality-enhanced expression images via a SR network. Specifically, a multi-stage attention-aware consistency loss module is introduced to help the SR network achieve discriminative feature restoration guided by attention information. Furthermore, a prediction consistency loss module is also developed to encourage the SR network to restore discriminative features by reducing the difference in prediction information between the restored and original normal-resolution images. Therefore, more accurate results are obtained by performing FER on the restored images. We conduct extensive experiments to demonstrate that the proposed low-resolution FER solution can help SR methods restore features favorable for FER while maintaining acceptable FER performance in various resolution degradation scenarios. The proposed method effectively improves the FER challenge under resolution degradation conditions, which is of good reference value for real-world applications.

Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognition

Facial expression recognition (FER) is one of the most important computer vision tasks for understanding human inner emotions. However, the poor generation ability of the FER model limits its applicability due to tremendous intraclass variation. Especially for expressions of varying intensities, the appearance differences among weak expressions are subtle, which makes FER tasks challenging. In response to these issues, this paper presents a dual subspace manifold learning method based on a graph convolutional network (GCN) for intensity-invariant FER tasks. Our method treats the target task as a node classification problem and learns the manifold representation using two subspace analysis methods: locality preserving projection (LPP) and peak-piloted locality preserving projection (PLPP). Inspired by the classic LPP, which maintains local similarity among data, this paper introduces a novel PLPP that maintains the locality between peak expressions and non-peak expressions to enhance the representation of weak expressions. This paper also reports two subspace fusion methods, one based on a weighted adjacency matrix and another on a self-attention mechanism, that combine the LPP and PLPP to further improve FER performance. The second method achieves a recognition accuracy of 93.83% on the CK+, 74.86% on the Oulu-CASIA and 75.37% on the MMI for weak expressions, outperforming state-of-the-art methods.

Towards a Better Performance in Facial Expression Recognition: A Data-Centric Approach.

Facial expression is the best evidence of our emotions. Its automatic detection and recognition are key for robotics, medicine, healthcare, education, psychology, sociology, marketing, security, entertainment, and many other areas. Experiments in the lab environments achieve high performance. However, in real-world scenarios, it is challenging. Deep learning techniques based on convolutional neural networks (CNNs) have shown great potential. Most of the research is exclusively model-centric, searching for better algorithms to improve recognition. However, progress is insufficient. Despite being the main resource for automatic learning, few works focus on improving the quality of datasets. We propose a novel data-centric method to tackle misclassification, a problem commonly encountered in facial image datasets. The strategy is to progressively refine the dataset by successive training of a CNN model that is fixed. Each training uses the facial images corresponding to the correct predictions of the previous training, allowing the model to capture more distinctive features of each class of facial expression. After the last training, the model performs automatic reclassification of the whole dataset. Unlike other similar work, our method avoids modifying, deleting, or augmenting facial images. Experimental results on three representative datasets proved the effectiveness of the proposed method, improving the validation accuracy by 20.45%, 14.47%, and 39.66%, for FER2013, NHFI, and AffectNet, respectively. The recognition rates on the reclassified versions of these datasets are 86.71%, 70.44%, and 89.17% and become state-of-the-art performance.

Parameter optimization in convolutional neural network for improving performance of facial expression recognition

The recognition of similar facial expressions presents a notable challenge, necessitating a focus on the parameters within the fundamental Convolutional Neural Network (CNN) architecture, which serves as a cornerstone in the field of image classification. This research endeavor aims to enhance the model's capacity for facial expression recognition by employing a controlled variable method to examine two specific parameters in a self-designed small CNN: the number of filters and convolutional layers. More specifically, while the filters were fixed at 3, the layers varied from 3 to 6 to 9. Similarly, as the number of the filters rose to 6, the number of the layers also incremented from 3 to 6 to 9. Furthermore, while the number of the filters reached 12, the number of the layers went from 3 to 6 to 9 too. Finally, with the filters increasing to 24, the layers rose from 3 to 6 to 9 as well. Experimental results indicate that both increasing the number of filters and convolutional layers can increment the performance of model in facial expression recognition. Furthermore, increasing the number of filters can exert a more prominent influence on improving the accuracy of facial expression recognition.

Robust facial expression recognition with Transformer Block Enhancement Module

Recently, facial expression recognition (FER) methods have achieved significant progress. However, FER is still challenged by factors such as uneven illumination and low-quality expression images. Exploring the potential of facial expression features to achieve robust FER is particularly important. Inspired by the Transformer’s excellent performance in modeling long-range dependencies and in vision tasks, this paper proposes a Transformer Block Enhancement Module (TBEM) for enhancing the feature representation of facial expressions. The proposed module contains a Channel Enhancement (CE) block and a Spatial Enhancement (SE) block. The CE block can adaptively enhance the expression features on the channel dimension by effectively leveraging the channel dependency information, while the SE block enhances the expression features on the spatial dimension by integrating the spatial dependency information. TBEM can output a more robust expression representation by combining CE and SE. The proposed artificial intelligence learning module greatly improves the recognition accuracy of FER engineering tasks. To further illustrate the application of TBEM in real-world FER engineering, three engineering problems are used for verification. Extensive experiments demonstrate that the proposed method improves FER performance by focusing on more accurate decisional features and can be easily embedded into regular convolutional neural network models to help them improve the accuracy on FER tasks by about 2.64%–3.03%. The results show the proposed method achieves accuracies of 90.57% on FERPlus, 89.41% on RAFDB basic and 68.43% on RAFDB compound, respectively. The proposed method also provides a meaningful reference for further research on applying Transformer to other tasks.

Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher

In this paper, we aim to improve the performance of in-the-wild Facial Expression Recognition (FER) by exploiting semi-supervised learning. Large-scale labeled data and deep learning methods have greatly improved the performance of image recognition. However, the performance of FER is still not ideal due to the lack of training data and incorrect annotations (e.g., label noises). Among existing in-the-wild FER datasets, reliable ones contain insufficient data to train robust deep models while large-scale ones are annotated in lower quality. To address this problem, we propose a semi-supervised learning algorithm named Progressive Teacher (PT) to utilize reliable FER datasets as well as large-scale unlabeled expression images for effective training. On the one hand, PT introduces semi-supervised learning method to relieve the shortage of data in FER. On the other hand, it selects useful labeled training samples automatically and progressively to alleviate label noise. PT uses selected clean labeled data for computing the supervised classification loss and unlabeled data for unsupervised consistency loss. Experiments on widely-used databases RAF-DB and FERPlus validate the effectiveness of our method, which achieves state-of-the-art performance with accuracy of 89.57% on RAF-DB. Additionally, when the synthetic noise rate reaches even 30%, the performance of our PT algorithm only degrades by 4.37%.

Symmetric Multi-Scale Residual Network Ensemble with Weighted Evidence Fusion Strategy for Facial Expression Recognition

To extract facial features with different receptive fields and improve the decision fusion performance of network ensemble, a symmetric multi-scale residual network (SMResNet) ensemble with a weighted evidence fusion (WEF) strategy for facial expression recognition (FER) was proposed. Firstly, aiming at the defect of connecting different filter groups of Res2Net only from one direction in a hierarchical residual-like style, a symmetric multi-scale residual (SMR) block, which can symmetrically extract the features from two directions, was improved. Secondly, to highlight the role of different facial regions, a network ensemble was constructed based on three networks of SMResNet to extract the decision-level semantic of the whole face, eyes, and mouth regions, respectively. Meanwhile, the decision-level semantics of three regions were regarded as different pieces of evidence for decision-level fusion based on the Dempster-Shafer (D-S) evidence theory. Finally, to fuse the different regional expression evidence of the network ensemble, which has ambiguity and uncertainty, a WEF strategy was introduced to overcome conflicts within evidence based on the support degree adjustment. The experimental results showed that the facial expression recognition rates achieved 88.73%, 88.46%, and 88.52% on FERPlus, RAF-DB, and CAER-S datasets, respectively. Compared with other state-of-the-art methods on three datasets, the proposed network ensemble, which not only focuses the decision-level semantics of key regions, but also addresses to the whole face for the absence of regional semantics under occlusion and posture variations, improved the performance of facial expression recognition in the wild.

Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition.

This paper presents a novel facial expression recognition network, called Distract your Attention Network (DAN). Our method is based on two key observations in biological visual perception. Firstly, multiple facial expression classes share inherently similar underlying facial appearance, and their differences could be subtle. Secondly, facial expressions simultaneously exhibit themselves through multiple facial regions, and for recognition, a holistic approach by encoding high-order interactions among local features is required. To address these issues, this work proposes DAN with three key components: Feature Clustering Network (FCN), Multi-head Attention Network (MAN), and Attention Fusion Network (AFN). Specifically, FCN extracts robust features by adopting a large-margin learning objective to maximize class separability. In addition, MAN instantiates a number of attention heads to simultaneously attend to multiple facial areas and build attention maps on these regions. Further, AFN distracts these attentions to multiple locations before fusing the feature maps to a comprehensive one. Extensive experiments on three public datasets (including AffectNet, RAF-DB, and SFEW 2.0) verified that the proposed method consistently achieves state-of-the-art facial expression recognition performance. The DAN code is publicly available.

Impact of Facial Landmark Localization on Facial Expression Recognition

Although facial landmark localization (FLL) approaches are becoming increasingly accurate for characterizing facial regions, one question remains unanswered: what is the impact of these approaches on subsequent related tasks? In this paper, the focus is put on facial expression recognition (FER), where facial landmarks are used for face registration, which is a common usage. Since the most used datasets for facial landmark localization do not allow for a proper measurement of performance according to the different difficulties (e.g., pose, expression, illumination, occlusion, motion blur), we also quantify the performance of recent approaches in the presence of head pose variations and facial expressions. Finally, a study of the impact of these approaches on FER is conducted. We show that the landmark accuracy achieved so far optimizing the conventional Euclidean distance does not necessarily guarantee a gain in performance for FER. To deal with this issue, we propose a new evaluation metric for FLL adapted to FER.

Geometry-Aware Facial Expression Recognition via Attentive Graph Convolutional Networks

Learning discriminative representations with good robustness from facial observations serves as a fundamental step towards intelligent facial expression recognition (FER). In this paper, we propose a novel geometry-aware FER framework to boost the FER performance based on both the geometric and appearance knowledge. Specifically, we propose an encoding strategy for facial landmarks, and adopt a graph convolutional network (GCN) to fully explore the structural information of the facial components behind different expressions. A convolutional neural network (CNN) is further applied to the whole facial observation to learn the global characteristics of different expressions. The features from these two networks are fused into a comprehensive high-semantic representation, which promotes the FER reasoning from both visual and structural perspectives. Moreover, to facilitate the networks to concentrate on the most informative facial regions and components, we introduce multi-level attention mechanisms into the proposed framework, which enhance the reliability of the learned representations for effective FER. Experiments on two challenging FER benchmarks demonstrate that the attentive graph-based learning on the facial geometry boosts the FER accuracy. Furthermore, the insensitivity of the geometric information to the appearance variations also improves the generalization of the proposed framework.

Typical Facial Expression Network Using a Facial Feature Decoupler and Spatial-Temporal Learning

Facial expression recognition (FER) accuracy is often affected by an individuals unique facial characteristics. Recognition performance can be improved if the influence from these physical characteristics is minimized. Using video instead of single image for FER provides better results but requires extracting temporal features and the spatial structure of facial expressions in an integrated manner. We propose a new network called Typical Facial Expression Network (TFEN) to address both challenges. TFEN uses two deep two-dimensional (2D) convolutional neural networks (CNNs) to extract facial and expression features from input video. A facial feature decoupler decouples facial features from expression features to minimize the influence from inter-subject face variations. These networks combine with a 3D CNN and form a spatial-temporal learning network to jointly explore the spatial-temporal features in a video. A facial recognition network works as an adversarial network to refine the facial feature decoupler and the network performance by minimizing the residual influence of facial features after decoupling. The whole network is trained with an adversarial algorithm to improve FER performance. TFEN was evaluated on four popular dynamic FER datasets. Experimental results show TFEN achieves or outperforms the recognition accuracy of state-of-the-art approaches.

Face expression recognition based on NGO-BILSTM model.

Facial expression recognition has always been a hot topic in computer vision and artificial intelligence. In recent years, deep learning models have achieved good results in accurately recognizing facial expressions. BILSTM network is such a model. However, the BILSTM network's performance depends largely on its hyperparameters, which is a challenge for optimization. In this paper, a Northern Goshawk optimization (NGO) algorithm is proposed to optimize the hyperparameters of BILSTM network for facial expression recognition. The proposed methods were evaluated and compared with other methods on the FER2013, FERplus and RAF-DB datasets, taking into account factors such as cultural background, race and gender. The results show that the recognition accuracy of the model on FER2013 and FERPlus data sets is much higher than that of the traditional VGG16 network. The recognition accuracy is 89.72% on the RAF-DB dataset, which is 5.45, 9.63, 7.36, and 3.18% higher than that of the proposed facial expression recognition algorithms DLP-CNN, gACNN, pACNN, and LDL-ALSG in recent 2 years, respectively. In conclusion, NGO algorithm effectively optimized the hyperparameters of BILSTM network, improved the performance of facial expression recognition, and provided a new method for the hyperparameter optimization of BILSTM network for facial expression recognition.

ATTENTION BASED CONVOLUTIONAL NEURAL NETWORK FOR FACIAL EXPRESSION RECOGNITION

Many deep learning-based methods have been suggested in recent years to improve facial expression recognition performance, but they are unable to extract the small facial changes in skin textures, such as wrinkles and furrows, which indicate changes in expression. We put forward an attention mechanism-based Convolutional Neural Network (CNN) for face expression recognition to get around this issue. The attention module, classification module feature extraction module and the reconstruction module make up the architectures four components. The feature extraction module, which consists of two distinct CNN processing streams-one for raw images and the other for Local Binary Pattern (LBP) feature maps. The LBP features extract image texture data before capturing the minute facial movements, which can enhance network efficiency. The neural network can pay more attention to useful characteristics by using an attention mechanism. To improve the attention model and produce improved results, we combine LBP features and convolutional features. We use the CK+ and Oulu-CASIA datasets to test the proposed methodology. The experimental findings show that the suggested method is workable and efficient. KEYWORDS: Convolution neural network, Local binary pattern, Facial expression recognition.

3-D Facial Feature Reconstruction and Learning Network for Facial Expression Recognition in the Wild

Facial expression recognition (FER) in the wild unavoidably suffers from the effects of face posture, illumination, and partial occlusion. In this article, we attempt to alleviate the above negative effects and improve the performance of FER in the wild based on 3-D face feature reconstruction and learning. Three-dimensional face reconstruction not only can effectively make up for the facial apparent information missing inform a 2-D face images but can also extract accurate 3-D facial geometric information in self-occlusion and extreme illumination scenarios. Therefore, we propose a novel end-to-end trainable 3-D face feature reconstruction and learning network (3-DF-RLN) is proposed for FER in the wild. In 3-DF-RLN, the 2-D implicitly frontalized face apparent data and 3-D facial landmarks are reconstructed by a 3-D face reconstruction module and input to two feature extraction pathways. The appearance pathway learns apparent features from the reconstructed 2-D face apparent data using a convolutional neural network. The geometry pathway learns the geometric features from the reconstructed 3-D facial landmarks using a graph convolutional network. Finally, FER is achieved via the fusion of the two pathways. Extensive experiments were conducted to evaluate the proposed method with three benchmark databases, including Multi-PIE, RAF-DB, and AffectNet. The results show that the proposed 3-DF-RLN model has better FER performance, both in the lab and in the wild. In addition, the face graph from the geometry pathway reveals the correlations between facial landmarks in FER.

Machine to brain: facial expression recognition using brain machine generative adversarial networks.

The human brain can effectively perform Facial Expression Recognition (FER) with a few samples by utilizing its cognitive ability. However, unlike the human brain, even the well-trained deep neural network is data-dependent and lacks cognitive ability. To tackle this challenge, this paper proposes a novel framework, Brain Machine Generative Adversarial Networks (BM-GAN), which utilizes the concept of brain's cognitive ability to guide a Convolutional Neural Network to generate LIKE-electroencephalograph (EEG) features. More specifically, we firstly obtain EEG signals triggered from facial emotion images, then we adopt BM-GAN to carry out the mutual generation of image visual features and EEG cognitive features. BM-GAN intends to use the cognitive knowledge learnt from EEG signals to instruct the model to perceive LIKE-EEG features. Thereby, BM-GAN has a superior performance for FER like the human brain. The proposed model consists of VisualNet, EEGNet, and BM-GAN. More specifically, VisualNet can obtain image visual features from facial emotion images and EEGNet can obtain EEG cognitive features from EEG signals. Subsequently, the BM-GAN completes the mutual generation of image visual features and EEG cognitive features. Finally, the predicted LIKE-EEG features of test images are used for FER. After learning, without the participation of the EEG signals, an average classification accuracy of 96.6 % is obtained on Chinese Facial Affective Picture System dataset using LIKE-EEG features for FER. Experiments demonstrate that the proposed method can produce an excellent performance for FER.

Hybrid Convolutional Neural Network and Long Short-Term Memory Approach for Facial Expression Recognition

Facial Expression Recognition (FER) has been an important field of research for several decades. Extraction of emotional characteristics is crucial to FERs, but is complex to process as they have significant intra-class variances. Facial characteristics have not been completely explored in static pictures. Previous studies used Convolution Neural Networks (CNNs) based on transfer learning and hyperparameter optimizations for static facial emotional recognitions. Particle Swarm Optimizations (PSOs) have also been used for tuning hyperparameters. However, these methods achieve about 92 percent in terms of accuracy. The existing algorithms have issues with FER accuracy and precision. Hence, the overall FER performance is degraded significantly. To address this issue, this work proposes a combination of CNNs and Long Short-Term Memories (LSTMs) called the HCNN-LSTMs (Hybrid CNNs and LSTMs) approach for FERs. The work is evaluated on the benchmark dataset, Facial Expression Recog Image Ver (FERC). Viola-Jones (VJ) algorithms recognize faces from preprocessed images followed by HCNN-LSTMs feature extractions and FER classifications. Further, the success rate of Deep Learning Techniques (DLTs) has increased with hyperparameter tunings like epochs, batch sizes, initial learning rates, regularization parameters, shuffling types, and momentum. This proposed work uses Improved Weight based Whale Optimization Algorithms (IWWOAs) to select near-optimal settings for these parameters using best fitness values. The experimental findings demonstrated that the proposed HCNN-LSTMs system outperforms the existing methods.