Facial Expression Recognition Method Research Articles

A classroom facial expression recognition method based on attention mechanism

Compared with other facial expression recognition, classroom facial expression recognition should pay more attention to the feature extraction of a specific region to reflect the attention of students. However, most features are extracted with complete facial images by deep neural networks. In this paper, we proposed a new expression recognition based on attention mechanism, where more attention would be paid in the channel information which have much relationship with the expression classification instead of depending on all channel information. A new classroom expression classification has also been concluded with considering the concentration. Moreover, activation function is modified to reduce the number of parameters and computations, at the same time, dropout regularization is added after the pool layer to prevent overfitting of the model. The experiments show that the accuracy of our method named Ixception has an maximize improvement of 5.25% than other algorithms. It can well meet the requirements of the analysis of classroom concentration.

PIDViT: Pose-Invariant Distilled Vision Transformer for Facial Expression Recognition in the Wild

Many Facial expression recognition methods have achieved great success, but they only considered front facial images or facial images close to the front. Besides, unlike in-the-laboratory datasets, the facial images in the real world (or in the wild) are without lighting and pose control, so that it is a big challenge to recognize these facial expressions. In this paper, the PIDViT (i.e., Pose-Invariant Distilled Vision Transformer) using the teacher-student model for the probability distributions of facial expressions of frontal and multi-pose faces was proposed and solved the pose variance and occlusion issues on expression recognition. First, the multi-pose face dataset FairFace-3D was built from the original FairFace and then used to train pose-invariance on the PIDViT. The PIDViT was trained in two stages; stage 1 is to train the PIDViT to achieve the consistency of facial expressions between frontal faces and multi-pose faces, and stage 2 is to use the student model pre-trained in stage 1 and train facial expressions further on target datasets. Finally, comprehensive experiments were conducted on three in the wild facial expression datasets, and the results validates the generalization of the PIDViT and its superiority over most state-of-the-art models.

Vision Transformer With Attentive Pooling for Robust Facial Expression Recognition

Facial Expression Recognition (FER) in the wild is an extremely challenging task. Recently, some Vision Transformers (ViT) have been explored for FER, but most of them perform inferiorly compared to Convolutional Neural Networks (CNN). This is mainly because the new proposed modules are difficult to converge well from scratch due to lacking inductive bias and easy to focus on the occlusion and noisy areas. TransFER, a representative transformer-based method for FER, alleviates this with multi-branch attention dropping but brings excessive computations. On the contrary, we present two attentive pooling (AP) modules to pool noisy features directly. The AP modules include Attentive Patch Pooling (APP) and Attentive Token Pooling (ATP). They aim to guide the model to emphasize the most discriminative features while reducing the impacts of less relevant features. The proposed APP is employed to select the most informative patches on CNN features, and ATP discards unimportant tokens in ViT. Being simple to implement and without learnable parameters, the APP and ATP intuitively reduce the computational cost while boosting the performance by ONLY pursuing the most discriminative features. Qualitative results demonstrate the motivations and effectiveness of our attentive poolings. Besides, quantitative results on six in-the-wild datasets outperform other state-of-the-art methods.

Transferable driver facial expression recognition based on joint discriminative correlation alignment network with enhanced feature attention

AbstractFacial expression is closely related to the emotions of drivers, thus facilitating safe driving detection in advanced driving assistance system (ADAS). Recently, deep learning techniques have become prevalent for facial expression recognition. However, for driving scenarios, the facial expression recognition is mainly challenged by the problems of small sample size as well as effective feature representation. To address the above issues, this paper puts forward a transfer learning‐based method for driver facial expression recognition by fully exploiting the other sources of facial expression data that may obey different distributions. Specifically, an enhanced feature attention module is firstly devised such that affluent features with multi‐scales can be extracted and refined based on the spatial and channel attention mechanisms. Then, a joint correlation alignment loss is presented ensuring that the samples in the source and target domains are transformed into the shared common subspace to reduce difference of both the marginal and conditional distributions. Multiple transfer learning tasks on real‐world data are carried out to evaluate the proposed method. The experimental results show that our model achieves better recognition accuracy for driver facial expression compared with several traditional and deep learning‐based transfer learning algorithms.

Investigation into facial expression recognition methods: a review

Facial expression recognition (FER) is a rapidly emerging topic in computer vision that has gotten a lot of interest because of its numerous applications in fields including psychology, sociology, human-computer interaction (HCI), and security. FER seeks to recognise and analyse human facial expressions in order to determine emotions and other mental states. Several strategies, including feature-based, kernel-based, and deep learning-based methods, have been developed and implemented in FER in recent years. FER’s major goal is to extract and identify the most discriminating elements that accurately represent the emotions expressed by facial expressions. The literature reviewed in this field shows that deep learning-based methods have outperformed traditional feature-based and kernel-based methods in terms of accuracy and robustness in recognizing facial expressions. However, these deep learning-based methods also pose several challenges, such as the need for large labeled-data-sets, robustness to different facial poses and illumination conditions, and generalization to unseen data. Despite these challenges, the field of FER is expected to continue growing, and future research will likely focus on addressing these challenges and improving the accuracy and robustness of FER systems.

Efficient Facial Expression Recognition With Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

The emotion perception capability could endow machines with deeper feedback evaluation of human–machine interaction (HMI), which makes the experience of HMI more harmonious and natural. Currently, how to obtain the robust surface and semantic facial representation features has not been fully explored, moreover, the parameters and computation of conventional convolutional neural network models, which are adopted in facial expression recognition (FER) are too heavy, it is not conducive to the deployment of the HMI system. Therefore, this article proposes a representation reinforcement network (RRN) and transfer self-training (TST) based efficient FER method. Our designed RRN is mainly composed of two modules, i.e., surface representation reinforcement (SurRR) module and semantic representation reinforcement (SemaRR) module. SurRR module fuses the facial features in different dimensions and highlights key feature messaging nodes in feature maps. SemaRR module establishes the deep semantic representation of global facial regions on spatial and channel dimensions. Therefore, RRN has stronger feature extraction capability when the parameters and computational complexity are significantly reduced, and TST is proposed for contributing to the training effect of RRN at the condition of insufficient training data. We tested our method on CK+, RaFD, FERPLUS, and RAF-DB datasets, and the recognition accuracy are 100%, 98.62%, 89.64%, and 88.72%, respectively. Furthermore, the preliminary application experiment, which demonstrates the feasibility of our method in practice during HMI.

An extensive study of facial expression recognition using artificial intelligence techniques with different datasets

<p>Machine and deep learning (DL) algorithms have advanced to a point where a wide range of crucial real-world computer vision problems can be solved. Facial Expression Recognition (FER) is one of these applications; it is the foremost non-verbal intentions and a fascinating study of symmetry. A prevalent application of deep learning has become the area of vision, where facial expression recognition has emerged as one of the most promising new frontiers. Latterly deep learning-based FER models have been plagued by technical problems, including under-fitting and over-fitting. Probably inadequate information is used for training and expressing ideas. With these considerations in mind, this article gives a systematic and complete survey of the most cutting-edge AI strategies and gives a conclusion to address the aforementioned problems. It is also a scheme of classification for existing facial proposals in compact. This survey analyses the structure of the usual FER method and discusses the feasible technologies that may be used in its respective elements. In addition, this study provides a summary of seventeen widely-used FER datasets that reviews functioning novel machine and DL networks suggested by academics and outline their benefits and liability in the context of facial expression acknowledgment based on static replicas. Finally, this study discusses the research obstacles and open consequences of that well-conditioned face expression recognition scheme.</p>

Lightweight Facial Expression Recognition Based on Class-Rebalancing Fusion Cumulative Learning

In the research of Facial Expression Recognition (FER), the inter-class of facial expression data is not evenly distributed, the features extracted by networks are insufficient, and the FER accuracy and speed are relatively low for practical applications. Therefore, a lightweight and efficient method based on class-rebalancing fusion cumulative learning for FER is proposed in our research. A dual-branch network (Regular feature learning and Rebalancing-Cumulative learning Network, RLR-CNet) is proposed, where the RLR-CNet uses the improvement in the lightweight ShuffleNet with two branches (feature learning and class-rebalancing) based on cumulative learning, which improves the efficiency of our model recognition. Then, to enhance the generalizability of our model and pursue better recognition efficiency in real scenes, a random masking method is improved to process datasets. Finally, in order to extract local detailed features and further improve FER efficiency, a shuffle attention module (SA) is embedded in the model. The results demonstrate that the recognition accuracy of our RLR-CNet is 71.14%, 98.04%, and 87.93% on FER2013, CK+, and RAF-DB, respectively. Compared with other FER methods, our method has great recognition accuracy, and the number of parameters is only 1.02 MB, which is 17.74% lower than that in the original ShuffleNet.

Multi-Relations Aware Network for In-the-Wild Facial Expression Recognition

Facial expression recognition (FER) becomes more challenging in the wild due to unconstrained conditions, such as the different illumination, pose changes, and occlusion of the face. Current FER methods deploy the attention mechanism in deep neural networks to improve the performance. However, these models only capture the limited attention features and relationships. Thus this paper proposes a novel FER framework called multi-relations aware network (MRAN), which can focus on global and local attention features and learn the multi-level relationships among local regions, between global-local features and among different samples, to obtain efficient emotional features. Specifically, our method first imposes the spatial attention on both the whole face and local regions to simultaneously learn the global and local salient features. After that, a region relation transformer is deployed to capture the internal structure among local facial regions, and a global-local relation transformer is designed to learn the fusion relations between global features and local features for different facial expressions. Subsequently, a sample relation transformer is deployed to focus on intrinsic similarity relationship among training samples, which promotes invariant feature learning for each expression. Finally, a joint optimization strategy is designed to efficiently optimize the model. The conducted experimental results on in-the-wild databases show that our method obtains the superior performance compared to some state-of-the-art models.

Robust facial expression recognition with Transformer Block Enhancement Module

Recently, facial expression recognition (FER) methods have achieved significant progress. However, FER is still challenged by factors such as uneven illumination and low-quality expression images. Exploring the potential of facial expression features to achieve robust FER is particularly important. Inspired by the Transformer’s excellent performance in modeling long-range dependencies and in vision tasks, this paper proposes a Transformer Block Enhancement Module (TBEM) for enhancing the feature representation of facial expressions. The proposed module contains a Channel Enhancement (CE) block and a Spatial Enhancement (SE) block. The CE block can adaptively enhance the expression features on the channel dimension by effectively leveraging the channel dependency information, while the SE block enhances the expression features on the spatial dimension by integrating the spatial dependency information. TBEM can output a more robust expression representation by combining CE and SE. The proposed artificial intelligence learning module greatly improves the recognition accuracy of FER engineering tasks. To further illustrate the application of TBEM in real-world FER engineering, three engineering problems are used for verification. Extensive experiments demonstrate that the proposed method improves FER performance by focusing on more accurate decisional features and can be easily embedded into regular convolutional neural network models to help them improve the accuracy on FER tasks by about 2.64%–3.03%. The results show the proposed method achieves accuracies of 90.57% on FERPlus, 89.41% on RAFDB basic and 68.43% on RAFDB compound, respectively. The proposed method also provides a meaningful reference for further research on applying Transformer to other tasks.

HELPFuL: Human Emotion Label Prediction Based on Fuzzy Learning for Realizing Artificial Intelligent in IoT

Human emotion label prediction is crucial to Artificial Intelligent in the Internet of Things (IoT). Facial expression recognition is the main technique to predict human emotion labels. Existing facial expression recognition methods do not consider the compound emotion and the fuzziness of emotion labels. Fuzzy learning is a mathematical tool for dealing with fuzziness and uncertainty information. The advantage of using fuzzy learning for human emotion recognition is that multiple fuzzy sentiment labels can be processed simultaneously. This paper proposes a fuzzy learning-based expression recognition method for human emotion label prediction. First, a fuzzy label distribution system is constructed using fuzzy sets for representing facial expressions. Then, two fuzzy label distribution prediction methods based on fuzzy rough sets are proposed to solve the compound emotion prediction. The probability that a sample is likely and definitely belongs to an emotion is obtained by calculating the upper and lower approximations. Experiments show the proposed algorithm not only performs well on human emotion label prediction but can also be used for other label distribution prediction tasks. The proposed method is more accurate and more general than other methods. The improvement of the method on the effect of emotion recognition extends the application scope of artificial intelligence in IoT.

Attack Can Benefit: An Adversarial Approach to Recognizing Facial Expressions under Noisy Annotations

The real-world Facial Expression Recognition (FER) datasets usually exhibit complex scenarios with coupled noise annotations and imbalanced classes distribution, which undoubtedly impede the development of FER methods. To address the aforementioned issues, in this paper, we propose a novel and flexible method to spot noisy labels by leveraging adversarial attack, termed as Geometry Aware Adversarial Vulnerability Estimation (GAAVE). Different from existing state-of-the-art methods of noisy label learning (NLL), our method has no reliance on additional information and is thus easy to generalize to the large-scale real-world FER datasets. Besides, the combination of Dataset Splitting module and Subset Refactoring module mitigates the impact of class imbalance, and the Self-Annotator module facilitates the sufficient use of all training data. Extensive experiments on RAF-DB, FERPlus, AffectNet, and CIFAR-10 datasets validate the effectiveness of our method. The stabilized enhancement based on different methods demonstrates the flexibility of our proposed GAAVE.

Designing a Dual-level Facial Expression Evaluation System for Performers Using Geometric Features and Petri Nets

The existing methods of Facial Expression Recognition (FER) primarily analyze six basic expressions, namely, surprise, happiness, anger, sadness, fear, and disgust. The Indian performing arts use three more well-defined expressions—peaceful, proud, and erotic. This study proposes an intelligent dual-level expression evaluation system that classifies performance-specific expressions into nine classes, assigns intensity level to the expression, and suggests modifications to the user for precise exhibition of an expression. At decision level-1 of a dual-level system, an 11-state model is designed to classify the nine expressions. The model is verified using the Colored Petri Net that helps analyze the rules used for the classification. Decision level-1 is also implemented using input feature database and SVM classifier, which yields 95.77% accuracy. Further, at decision level-2, SVM is used to assign an intensity level to the correctly classified images. In case of incorrectly exhibited expressions, feedback is provided to the user about the incorrect facial component state. The application-specific image dataset is used for the present study. The qualitative comparison with the other FER approaches is also carried out. With the increasing popularity of Indian classical dance in Western and Asian countries, the dual-level system enables learners of performing arts to practice, evaluate, and improvise their expression skills.

A survey of facial expression recognition in the wild

With the continuous development of facial expression technology, especially the development of deep learning, and the establishment of in-the-wild datesets in recent years, field face recognition has become a hot research field in the wild of facial expression recognition. Different from the traditional facial expression recognition(FER), in-the-wild facial facial expression recognition, the problem of recognition accuracy caused by illumination, occlusion or low image resolution. Therefore, in order to solve these problems, new methods have been put forward continuously in recent years. In this paper, we first continue to summarize the widely used datesets, and then summarize the paper methods of facial expression recognition in the field proposed in the past two years.

Emotional Analysis using Deep Learning

Emotions are mental states that accompany physiological changes in the face, resulting in facial expressions. Sympathy, anger, worry, joy, fright, and other significant emotions are a few examples. Facial expressions play a significant role in non-verbal communication because they encapsulate a person's emotions. There has been a great deal of research done on computer modelling of human emotions. Computer modelling of human emotions has been made possible by computer technology. However, it is still in its infancy. The authors attempted to overcome limitations and create new opportunities as well as gain a better understanding and implement this simple form of human interaction in proposed computer-aided world. It has been made possible to evaluate and interpret genuine facial expressions in real time thanks to new techniques for collecting facial expressions and quick, highresolution pictures. The FER (Facial Expression Recognition) method currently relies on motionless frames, which makes it very hard to recognize foreground from background in the absence of motion information. This study describes a real-time facial expression identification system that detects faces using HAAR cascading classification and classifies facial expressions using convolutional neural networks. The system utilizes a webcam to dynamically display emotion text and accurately categorizes seven major emotions, including anger, disgust, fear, happiness, sadness, surprise, and neutrality. Real-time facial expression recognition may be utilised in a number of real-world applications, including as airport security, trade, and medical monitoring.

A Multi-View Face Expression Recognition Method Based on DenseNet and GAN

Facial expression recognition (FER) techniques can be widely used in human-computer interaction, intelligent robots, intelligent monitoring, and other domains. Currently, FER methods based on deep learning have become the mainstream schemes. However, these methods have some problems, such as a large number of parameters, difficulty in being applied to embedded processors, and the fact that recognition accuracy is affected by facial deflection. To solve the problem of a large number of parameters, we propose a DSC-DenseNet model, which improves the standard convolution in DenseNet to depthwise separable convolution (DSC). To solve the problem wherein face deflection affects the recognition effect, we propose a posture normalization model based on GAN: a GAN with two local discriminators (LD-GAN) that strengthen the discriminatory abilities of the expression-related local parts, such as the parts related to the eyes, eyebrows, mouth, and nose. These discriminators improve the model’s ability to retain facial expressions and evidently benefits FER. Quantitative and qualitative experimental results on the Fer2013 and KDEF datasets have consistently shown the superiority of our FER method when working with multi-pose face images.

Hybrid Domain Consistency Constraints-Based Deep Neural Network for Facial Expression Recognition.

Facial expression recognition (FER) has received increasing attention. However, multiple factors (e.g., uneven illumination, facial deflection, occlusion, and subjectivity of annotations in image datasets) probably reduce the performance of traditional FER methods. Thus, we propose a novel Hybrid Domain Consistency Network (HDCNet) based on a feature constraint method that combines both spatial domain consistency and channel domain consistency. Specifically, first, the proposed HDCNet mines the potential attention consistency feature expression (different from manual features, e.g., HOG and SIFT) as effective supervision information by comparing the original sample image with the augmented facial expression image. Second, HDCNet extracts facial expression-related features in the spatial and channel domains, and then it constrains the consistent expression of features through the mixed domain consistency loss function. In addition, the loss function based on the attention-consistency constraints does not require additional labels. Third, the network weights are learned to optimize the classification network through the loss function of the mixed domain consistency constraints. Finally, experiments conducted on the public RAF-DB and AffectNet benchmark datasets verify that the proposed HDCNet improved classification accuracy by 0.3-3.84% compared to the existing methods.

DA-FER: Domain Adaptive Facial Expression Recognition

Facial expression recognition (FER) is an important field in computer vision with many practical applications. However, one of the challenges in FER is dealing with small sample data, where the number of samples available for training machine learning algorithms is limited. To address this issue, a domain adaptive learning strategy is proposed in this paper. The approach uses a public dataset with sufficient samples as the source domain and a small sample dataset as the target domain. Furthermore, the maximum mean discrepancy with kernel mean embedding is utilized to reduce the disparity between the source and target domain data samples, thereby enhancing expression recognition accuracy. The proposed Domain Adaptive Facial Expression Recognition (DA-FER) method integrates the SSPP module and Slice module to fuse expression features of different dimensions. Moreover, this method retains the regions of interest of the five senses to accomplish more discriminative feature extraction and improve the transfer learning capability of the network. Experimental results indicate that the proposed method can effectively enhance the performance of expression recognition. Specifically, when the self-collected Selfie-Expression dataset is used as the target domain, and the public datasets RAF-DB and Fer2013 are used as the source domain, the performance of expression recognition is improved to varying degrees, which demonstrates the effectiveness of this domain adaptive method.

Facial Expression Recognition with Contrastive Learning and Uncertainty-Guided Relabeling

Facial expression recognition (FER) plays a vital role in the field of human-computer interaction. To achieve automatic FER, various approaches based on deep learning (DL) have been presented. However, most of them lack for the extraction of discriminative expression semantic information and suffer from the problem of annotation ambiguity. In this paper, we propose an elaborately designed end-to-end recognition network with contrastive learning and uncertainty-guided relabeling, to recognize facial expressions efficiently and accurately, as well as to alleviate the impact of annotation ambiguity. Specifically, a supervised contrastive loss (SCL) is introduced to promote inter-class separability and intra-class compactness, thus helping the network extract fine-grained discriminative expression features. As for the annotation ambiguity problem, we present an uncertainty estimation-based relabeling module (UERM) to estimate the uncertainty of each sample and relabel the unreliable ones. In addition, to deal with the padding erosion problem, we embed an amending representation module (ARM) into the recognition network. Experimental results on three public benchmarks demonstrate that our proposed method facilitates the recognition performance remarkably with 90.91% on RAF-DB, 88.59% on FERPlus and 61.00% on AffectNet, outperforming current state-of-the-art (SOTA) FER methods. Code will be available at http//github.com/xiaohu-run/fer_supCon .

Learning transferable non-negative feature representation for facial expression recognition

Conventional facial expression recognition (FER) methods often assume that the training and test procedures are carried out on a single database, without considering the cross-database problem due to the difference in ages, collecting devices, nationalities, etc. To tackle the above challenge, we propose a novel transferable non-negative feature representation (TNFR) method in this work. The core ideas of TNFR are two-fold. First, we utilize the basis matrix as a transformation matrix to reduce the distribution discrepancy between two databases. Second, we utilize the label information to improve the discrimination ability of the model. In addition, we replace the original label matrix with a block-label matrix to make the model more flexible. Experiments on four benchmark databases are provided to demonstrate the effectiveness of the proposed TNFR method in comparison with state-of-the-art methods.