Expression Recognition Method Research Articles

FERDCNN: an efficient method for facial expression recognition through deep convolutional neural networks

Facial expression recognition (FER) has caught the research community’s attention recently because it can affect many real-life applications. Multiple studies have focused on automatic FER, most of which use a machine learning methodology, FER has continued to be a difficult and exciting issue in computer vision. Deep learning has recently drawn increased attention as a solution to several practical issues, including facial expression recognition. This article introduces an efficient method for FER (FERDCNN) verified on five different pre-trained deep CNN (DCNN) models (AlexNet, GoogleNet, ResNet-18, ResNet-50, and ResNet-101). In the proposed method, firstly the input image has been pre-processed using face detection, resizing, gamma correction, and histogram equalization techniques. Secondly, the images go through DCNN to extract deep features. Finally, support vector machine (SVM) and transfer learning are used to classify generated features. Recent methods have been employed to evaluate and contrast the performance of the proposed approach on two publicly standard databases namely, CK+ and JAFFE on the seven classes of fundamental emotions, including anger, disgust, fear, happiness, sadness, and surprise beside neutrality for CK+ and contempt for JAFFE. The suggested method tested Four different traditional supervised classifiers with deep features, Experimental found that AlexNet excels as a feature extractor, while SVM demonstrates superiority as a classifier because of this combination achieving the highest accuracy rates of 99.0% and 95.16% for the CK+ database and the JAFFE datasets, respectively.

Online classroom student engagement analysis based on facial expression recognition using enhanced YOLOv5 for mitigating cyberbullying

Abstract The rapid expansion of online education has heightened concerns about cyberbullying in virtual classrooms. This paper presents a comprehensive approach for detecting cyberbullying by analyzing students' engagement and emotional responses in online classrooms. Due to the influence of camera resolution and surrounding light in online classrooms, students' facial expressions are often blurry, and the changes in facial features may not be significant. Moreover, most current recognition algorithms utilize larger models, which may not be suitable for real-time detection in online environments. To address these challenges, this study introduces a student facial expression recognition (SFER) method based on an enhanced YOLOv5 (you only look once version 5) model, termed SFER-YOLOv5. Firstly, the improved Soft-NMS is employed to replace the original non-maximum suppression (NMS), effectively enhancing training efficiency. Then, the coordinate attention (CA) module is incorporated into the backbone network to improve detection accuracy, particularly in classroom settings with multiple students or when students are at a considerable distance from the camera. Next, the efficient intersection over union (EIoU) loss function is utilized. EIoU calculates width and height losses separately based on complete IoU (CIoU), replacing the aspect ratio. Finally, Focal Loss is introduced to address sample imbalance issues. The comparative results show that SFER-YOLOv5 achieves an mAP@0.5 of 78.4% on the FER-2013 dataset, 98.1% on the CK+ dataset, and 88.9% on our self-constructed dataset (SFEC). These results underscore the effectiveness of SFER-YOLOv5 in enhancing the accuracy of student facial expression recognition. The proposed method detects reduced engagement, offering a preventive strategy for mitigating cyberbullying in virtual learning environments.

Facial expression recognition based on convolutional network attention mechanism

Aiming at the problems of large number of parameters and weak recognition ability in expression recognition, a method of facial expression recognition based on convolutional network is proposed. An improved residual module is introduced to reduce the number of parameters while enhancing the attention to the expression area; the channel-space attention mechanism is used to realize the weight distribution of different dimensions and positions of the expression area extracted by the network, focusing on the subtle difference feature information in the key points of the expression; the detail module is used to further extract the deep feature information. In order to obtain higher accuracy, a joint loss function is introduced to increase the out-of-class distance and reduce the intra-class distance to improve the accuracy of expression recognition. This paper applies this network to the datasets FER2013 and CK+. The experimental results show that the average recognition rate of this algorithm is respectively and 63.91%，97.98%the number of parameters is 11.34 M. Compared with VGG network, residual network, etc., this model not only improves the recognition rate but also reduces the redundant parameters.

A New Joint Training Method for Facial Expression Recognition with Inconsistently Annotated and Imbalanced Data

Facial expression recognition (FER) plays a crucial role in various applications, including human–computer interaction and affective computing. However, the joint training of an FER network with multiple datasets is a promising strategy to enhance its performance. Nevertheless, widespread annotation inconsistencies and class imbalances among FER datasets pose significant challenges to this approach. This paper proposes a new multi-dataset joint training method, Sample Selection and Paired Augmentation Joint Training (SSPA-JT), to address these challenges. SSPA-JT models annotation inconsistency as a label noise problem and selects clean samples from auxiliary datasets to expand the overall dataset size while maintaining consistent annotation standards. Additionally, a dynamic matching algorithm is developed to pair clean samples of the tail class with noisy samples, which enriches the tail classes with diverse background information. Experimental results demonstrate that SSPA-JT achieved superior or comparable performance compared with the existing methods by addressing both annotation inconsistencies and class imbalance during multi-dataset joint training. It achieved state-of-the-art performance on RAF-DB and CAER-S datasets with accuracies of 92.44% and 98.22%, respectively, reflecting improvements of 0.2% and 3.65% over existing methods.

Reliability and validity of facial expression recognition in the screening of generalized anxiety disorder

IntroductionAnxiety disorder is one of the most prevalent mental disorders in China. However, there are obvious subjective factors in the current assessment of anxiety disorders, which may lead to certain diagnostic errors. The identification and diagnosis of anxiety disorders can be further improved if objective biological indicators are added in the assessment process. The current research validates facial expression recognition as a screening tool to assist in detecting generalized anxiety disorder. MethodsBased on the International Affective Picture System, we constructed an aided diagnostic experimental paradigm and recorded their facial expression. The split-half reliability was displayed by the Pearson correlation heatmap. The paradigm, GAD-7 and HAMA scales were administered to 60 generalized anxiety disorder patients and 60 matched healthy controls to evaluate the criterion-related validity. Additionally, we conducted a diagnostic study by using MINI as a gold standard and calculated ROC analysis to examine the screening performance of the facial expressions. ResultsThe heatmap showed very high correlations (r > 0.60, PS < 0.05) along the diagonal of the square heatmap (from the bottom left corner to the top right). The Pearson correlation coefficients between the GAD-7, HAMA and seven facial expressions ranged from −0.35(neutral, P < 0.01) to 0.34(angry, P < 0.01). The intergroup effects of neutral, anger and fear emotions were statistically significant (F = 18.893, P < 0.001; F = 20.535, P < 0.001; F = 9.091, P = 0.003). ROC analysis showed AUC for neutral, angry and scared facial expressions were 0.723, 0.792 and 0.727 respectively. ConclusionThis study constructed a tool for auxiliary screening of GAD patients and provided an objective automatic facial expression recognition method to assist psychological diagnosis.

Cutting-Edge Methods in Visage Expression Recognition and Ocular Tracking for Analyzing Remote Video Interviews

This paper focuses to augment an in the flesh image and video processor, capacitated with an artificial intelligence (AI) arbitrator that can speculate a job interviewee’s behavioral adeptness according to the facial dictions or expressions. We put forward a sentiment analysis model using machine learning and CNN for the reinforcement of decision-making in the job interview process. This is bought up by histogram of oriented gradients and support vector machine (HOG-SVM) with the addition of convolutional neural network (CNN) recognition in real-time video recorded interviews. The goal of applying this technology, is to develop a method that could automatically decode a candidate’s behavior by his or her facial language (or micro expressions) based on the behavioral ecology view of facial displays (BECV) which is different from the classical perspective of recognizing emotional states. This paper dispenses a trailblazing technique of determining the performance of a candidate in a video interview. To do this, a delineation of the analysis of sentiments and eye tracking technique study is intricated in which the results can be processed on a single screen to select the right person for the hire. Key Words: Artificial Intelligence(AI), Histogram of Oriented Gradients(HOG), Support Vector Machine(SVM), Convolutional Neural Network(CNN), Behavioral Ecology View of Facial Displays, Eye Tracking Technique, AVI(Automatic Visual Inspection).

TriCAFFNet: A Tri-Cross-Attention Transformer with a Multi-Feature Fusion Network for Facial Expression Recognition.

In recent years, significant progress has been made in facial expression recognition methods. However, tasks related to facial expression recognition in real environments still require further research. This paper proposes a tri-cross-attention transformer with a multi-feature fusion network (TriCAFFNet) to improve facial expression recognition performance under challenging conditions. By combining LBP (Local Binary Pattern) features, HOG (Histogram of Oriented Gradients) features, landmark features, and CNN (convolutional neural network) features from facial images, the model is provided with a rich input to improve its ability to discern subtle differences between images. Additionally, tri-cross-attention blocks are designed to facilitate information exchange between different features, enabling mutual guidance among different features to capture salient attention. Extensive experiments on several widely used datasets show that our TriCAFFNet achieves the SOTA performance on RAF-DB with 92.17%, AffectNet (7 cls) with 67.40%, and AffectNet (8 cls) with 63.49%, respectively.

Attentional visual graph neural network based facial expression recognition method

Attentional visual graph neural network based facial expression recognition method

A novel facial expression recognition model based on harnessing complementary features in multi-scale network with attention fusion

This paper presents a novel method for facial expression recognition using the proposed feature complementation and multi-scale attention model with attention fusion (FCMSA-AF). The proposed model consists of four main components: the shallow feature extractor module, parallel structured two-branch multi-scale attention module (MSA), feature complementing module (FCM), and attention fusion and classification module. The MSA module contains multi-scale attention modules in a cascaded fashion in two paths to learn diverse features. The upper and lower paths use left and right multi-scale blocks to extract and aggregate the features at different receptive fields. The attention networks in MSA focus on salient local regions to extract features at granular levels. The FCM uses the correlation between the feature maps in two paths to make the multi-scale attention features complementary to each other. Finally, the complementary features are fused through an attention network to form an informative holistic feature which includes subtle, visually varying regions in similar classes. Hence, complementary and informative features are used in classification to minimize information loss and capture the discriminating finer aspects of facial expression recognition. Experimental evaluation of the proposed model carried out on AffectNet and CK+ datasets achieve accuracies of 64.59% and 98.98%, respectively, outperforming some of the state-of-the-art methods.

Enhanced Hybrid Vision Transformer with Multi-Scale Feature Integration and Patch Dropping for Facial Expression Recognition.

Convolutional neural networks (CNNs) have made significant progress in the field of facial expression recognition (FER). However, due to challenges such as occlusion, lighting variations, and changes in head pose, facial expression recognition in real-world environments remains highly challenging. At the same time, methods solely based on CNN heavily rely on local spatial features, lack global information, and struggle to balance the relationship between computational complexity and recognition accuracy. Consequently, the CNN-based models still fall short in their ability to address FER adequately. To address these issues, we propose a lightweight facial expression recognition method based on a hybrid vision transformer. This method captures multi-scale facial features through an improved attention module, achieving richer feature integration, enhancing the network's perception of key facial expression regions, and improving feature extraction capabilities. Additionally, to further enhance the model's performance, we have designed the patch dropping (PD) module. This module aims to emulate the attention allocation mechanism of the human visual system for local features, guiding the network to focus on the most discriminative features, reducing the influence of irrelevant features, and intuitively lowering computational costs. Extensive experiments demonstrate that our approach significantly outperforms other methods, achieving an accuracy of 86.51% on RAF-DB and nearly 70% on FER2013, with a model size of only 3.64 MB. These results demonstrate that our method provides a new perspective for the field of facial expression recognition.

Facial expression recognition, a predictive tool for perceiving urban open space environments under audio-visual interaction

The quality of soundscape in urban open spaces is important to wellbeing. An AI-based facial expression recognition method has the potential to be a useful tool for predicting urban sound perception since it can correctly and in real time gather human perception data from facial photos. The purpose of this study is to provide classification criteria, time dimension data processing and indicator screening methods for facial expression recognition-based soundscape prediction. The study captured typical urban audiovisual environments and recorded facial expression data from subjects in a laboratory setting. The results of the study showed that there were significant differences in the effects of the three types of visual environments and the three types of acoustic environments on the subjects’ facial expressions, and that the effect of acoustic environments on visual perception (η2 0.067 on average) was generally greater than the effect of visual environments on acoustic perception (η2 0.035 on average). A linear regression model for predicting acoustic perception was developed based on audiovisual interactions, dynamic change patterns, and indicator selection. The results of this paper help to realize another perceptual dimension in the establishment of smart city and help achieve sustainability.

Dual-consistency constraints network for noisy facial expression recognition

Although existing facial expression recognition (FER) methods have achieved great success, their performance degrades significantly under noisy labels caused by low-quality images, ambiguous expressions, and subjective and incorrect labeling. Recent studies have shown that deep neural networks (DNNs) can easily overfit noisy labels, which poses a great challenge to FER task in real-world scenarios. To address this issue, we propose a novel Dual-consistency Constraints Network (DC-Net) to automatically suppress noisy samples during training. Specifically, we first propose a Class Activation Mapping (CAM) Attention Consistency (CAC), which makes the model focus on partially important feature information. As a result, we obtain more robust local feature representations and reduce excessive attention to noisy labels. Then, a Class Feature Consistency (CFC) is designed to encourage the model to focus on the global semantic information of the image. Finally, with the collaboration of the CAC and the CFC, DC-Net can learn robust local and global feature information to prevent the model from learning biased information with noisy labels. We conducted extensive experiments on three field datasets, including RAF-DB, AffectNet, and FERPlus2013. Experimental results show that DC-Net significantly outperforms state-of-the-art noisy labeling methods at different noise rates and generalizes well to other tasks with a large number of classes, such as CIFAR100 and Tiny- ImageNet.

Eye Strain Expression Classification using Attention Capsule Network for Adapting Screen Vision

Beside the conventional facial expression recognition methods, the research focuses on developing a system for recognizing various eye expressions under different screen conditions. This research deals with the use of Capsule Network (a recent Deep Learning algorithm) to enhance facial expression recognition capabilities and to develop adaptive screen technologies aimed at mitigating digital eye strain. The main objective of this research is to engineer a sophisticated system that employs the capabilities of Capsule Nets to recognize the various expressions that user makes and based on the recognized expression, dynamically modify screen settings, ensuring optimal user visual comfort. The research primarily concentrates on the exploration and application of various Capsule Net architectures designed for the recognition of expressions related to eye strain. The baseline model utilized elementary convolutional layers which feed into subsequent fully connected layers for the task of classification. The model has since been refined by incorporating advanced techniques such as attention mechanisms and more sophisticated network architectures where the classification is done by Capsule Network. Results have demonstrated a modest enhancement in the Capsule Net’s predictive performance, attributed to its superior spatial and hierarchical processing of facial features, in comparison to conventional deep learning approaches. The final model has an accuracy of 82.27%. As a final system the model has been deployed to an application to process frames from video camera in the device and make prediction to prompt the notifications or recommendations.

XTime: A general rule-based method for time expression recognition and normalization

Time expression (a.k.a., timex) recognition and normalization (TERN) is a crucial task for downstream research. However, previous studies have overlooked the critical characteristics of timexes that significantly impact the task. To gain deeper insights, we conduct an analysis across four diverse English datasets to examine the key attributes of timex constituents. Our analysis reveals several noteworthy observations, such as: timexes tend to very short; the majority of timexes contain time tokens; there exist strong mapping relationships between time tokens and timex types; there exists a priority relationship among timex types; and timex values exhibit only some standard formats. Based on these insights, we propose a novel general rule-based method termed XTime11Source codes and datasets are available at https://github.com/xszhong/XTime. to recognize timexes from free text and normalize them into standard formats. Notably, XTime’s rules are designed in a general and heuristic manner, enabling its independence of diverse domains and text types. Experimental evaluations conducted on both in-domain and out-of-domain English datasets demonstrate that XTime consistently outperforms or performs comparably to representative state-of-the-art methods.

Auto Diagnosis of Parkinson's Disease Via a Deep Learning Model Based on Mixed Emotional Facial Expressions.

Parkinson's disease (PD) is a common degenerative disease of the nervous system in the elderly. The early diagnosis of PD is very important for potential patients to receive prompt treatment and avoid the aggravation of the disease. Recent studies have found that PD patients always suffer from emotional expression disorder, thus forming the characteristics of "masked faces". Based on this, we thus propose an auto PD diagnosis method based on mixed emotional facial expressions in the paper. Specifically, the proposed method is cast into four steps: Firstly, we synthesize virtual face images containing six basic expressions (i.e., anger, disgust, fear, happiness, sadness, and surprise) via generative adversarial learning, in order to approximate the premorbid expressions of PD patients; Secondly, we design an effective screening scheme to assess the quality of the above synthesized facial expression images and then shortlist the high-quality ones; Thirdly, we train a deep feature extractor accompanied with a facial expression classifier based on the mixture of the original facial expression images of the PD patients, the high-quality synthesized facial expression images of PD patients, and the normal facial expression images from other public face datasets; Finally, with the well-trained deep feature extractor, we thus adopt it to extract the latent expression features for six facial expression images of a potential PD patient to conduct PD/non-PD prediction. To show real-world impacts, we also collected a new facial expression dataset of PD patients in collaboration with a hospital. Extensive experiments are conducted to validate the effectiveness of the proposed method for PD diagnosis and facial expression recognition.

POST: Prototype‐oriented similarity transfer framework for cross‐domain facial expression recognition

AbstractFacial expression recognition (FER) is one of the popular research topics in computer vision. Most deep learning expression recognition methods perform well on a single dataset, but may struggle in cross‐domain FER applications when applied to different datasets. FER under cross‐dataset also suffers from difficulties such as feature distribution deviation and discriminator degradation. To address these issues, we propose a prototype‐oriented similarity transfer framework (POST) for cross‐domain FER. The bidirectional cross‐attention Swin Transformer (BCS Transformer) module is designed to aggregate local facial feature similarities across different domains, enabling the extraction of relevant cross‐domain features. The dual learnable category prototypes is designed to represent potential space samples for both source and target domains, ensuring enhanced domain alignment by leveraging both cross‐domain and specific domain features. We further introduce the self‐training resampling (STR) strategy to enhance similarity transfer. The experimental results with the RAF‐DB dataset as the source domain and the CK+, FER2013, JAFFE and SFEW 2.0 datasets as the target domains, show that our approach achieves much higher performance than the state‐of‐the‐art cross‐domain FER methods.

Lightweight facial expression estimation for mobile computing in portable device

AbstractFacial expression recognition has been studied for many years, especially with the development of deep learning. However, the existing researches still have the following two issues. Firstly, the intensity of facial expression is neglected. Secondly, the deep learning based approaches cannot be directly deployed in the devices with limited resources. In order to tackle these two issues, this paper proposes a lightweight facial expression estimation method using a shallow ordinal regression algorithm, which is deployed in a portable smart device for mobile computing in IoTs. Compared with classification based facial expression recognition methods, ordinal regression considers the intensity of facial expression to achieve better mean absolute error (MAE), which is validated by experiments on several public facial expression datasets. The simulation in portable device also demonstrates its effectiveness for mobile computing.

CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition

Occlusion or posture change of the face in natural scenes has typical asymmetry; however, an asymmetric face plays a key part in the lack of information available for facial expression recognition. To solve the problem of low accuracy of asymmetric facial expression recognition, this paper proposes a fusion of channel global features and a spatial local information expression recognition network called the “Channel–Spatial Integration Network” (CSINet). First, to extract the underlying detail information and deepen the network, the attention residual module with a redundant information filtering function is designed, and the backbone feature-extraction network is constituted by module stacking. Second, considering the loss of information in the local key area of face occlusion, the channel–spatial fusion structure is constructed, and the channel features and spatial features are combined to enhance the accuracy of occluded facial recognition. Finally, before the full connection layer, more local spatial information is embedded into the global channel information to capture the relationship between different channel–spatial targets, which improves the accuracy of feature expression. Experimental results on the natural scene facial expression data sets RAF-DB and FERPlus show that the recognition accuracies of the modeling approach proposed in this paper are 89.67% and 90.83%, which are 13.24% and 11.52% higher than that of the baseline network ResNet50, respectively. Compared with the latest facial expression recognition methods such as CVT, PACVT, etc., the method in this paper obtains better evaluation results of masked facial expression recognition, which provides certain theoretical and technical references for daily facial emotion analysis and human–computer interaction applications.

A novel deep learning model to improve the recognition of students’ facial expressions in online learning environments

With the fast development of artificial intelligence and emerging technologies, automatic recognition of students’ facial expressions has received increased attention. Facial expressions are a kind of external manifestation of emotional states. It is important for teachers to assess students’ emotional states and adjust teaching activities accordingly. However, existing methods for automatic facial expression recognition have the limitations of low accuracy of recognition and poor feature extraction. To address the problem, this study proposed a novel deep learning model called DenseNetX-CBAM to improve facial expression recognition by utilizing a variant of densely connected convolutional networks (DenseNet) to reduce unnecessary parameters and strengthen the reuse of expression features between networks; moreover, convolutional block attention module (CBAM) was integrated to allow the networks to focus on important special regions and important channels when representing features. The proposed model was tested using 217 video clips of 33 students in an online course. The results demonstrated promising effects of the method in improving the accuracy of facial expression recognition, which can help teachers to accurately recognize students’ emotions and provide real-time adjustment in online learning environments.

Studies Advanced in Development and Application of Facial Expression Recognition

Facial expression is an important way to convey people's inner emotional changes, as the basis of emotional understanding and the prerequisite for computers to understand human emotions, expression recognition has attracted more and more research attention from academia and industry in recent years. Most of the early expression recognition methods relied on manual features such as texture, geometry, and contour. Thanks to the powerful feature representation capability of convolutional neural networks, deep learning-based face recognition technology has made breakthroughs in recognition accuracy and speed. This paper details the latest research progress in the field of face expression recognition, including the design ideas, key steps, advantages and disadvantages of representative methods. This paper also introduces the common face expression recognition datasets and quantitatively compares the results of different methods on these datasets. Finally, this paper discusses the problems in the face expression recognition research field and look forward to the future development.