Facial Expression Datasets Research Articles

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement.

In smart classroom environments, accurately recognizing students' facial expressions is crucial for teachers to efficiently assess students' learning states, timely adjust teaching strategies, and enhance teaching quality and effectiveness. In this paper, we propose a student facial expression recognition model based on multi-scale and deep fine-grained feature attention enhancement (SFER-MDFAE) to address the issues of inaccurate facial feature extraction and poor robustness of facial expression recognition in smart classroom scenarios. Firstly, we construct a novel multi-scale dual-pooling feature aggregation module to capture and fuse facial information at different scales, thereby obtaining a comprehensive representation of key facial features; secondly, we design a key region-oriented attention mechanism to focus more on the nuances of facial expressions, further enhancing the representation of multi-scale deep fine-grained feature; finally, the fusion of multi-scale and deep fine-grained attention-enhanced features is used to obtain richer and more accurate facial key information and realize accurate facial expression recognition. The experimental results demonstrate that the proposed SFER-MDFAE outperforms the existing state-of-the-art methods, achieving an accuracy of 76.18% on FER2013, 92.75% on FERPlus, 92.93% on RAF-DB, 67.86% on AffectNet, and 93.74% on the real smart classroom facial expression dataset (SCFED). These results validate the effectiveness of the proposed method.

Improved Histogram of Oriented Gradient (HOG) Feature Extraction for Facial Expressions Classification

Facial expression classification system is one of the implementations of machine learning (ML) that takes facial expression datasets, undergoes training, and then utilizes the trained results to recognize facial expressions in new facial images. The recognized facial expressions include anger, contempt, disgust, fear, happy, sadness, and surprise expressions. The method employed for facial feature extraction utilizes histogram-oriented gradient (HOG). This study proposes an enhancement method for HOG feature extraction by reducing the feature dimension into multiple sub-features based on gradient orientation intervals, referred to as HOG channel (HOG-C). Classifier testing techniques are divided into two methods for comparison—support vector machines (SVM) with HOG features and SVM with HOG-C features. The testing results demonstrate that SVM with HOG achieves an accuracy of 99.9% with an average training time of 18.03 minutes, while SVM with HOG-C attains a 100% accuracy with an average training time of 18.09 minutes. The testing outcomes reveal that the implementation of SVM with HOG-C successfully enhances accuracy for facial expression classification.

Advancing emotion recognition in facial expressions through PCA, RFE, and MLP Integration

Emotion recognition through facial expressions is vital for enhancing human-computer interaction, making systems more intuitive and responsive to user needs. This study introduces an innovative approach to emotion detection, leveraging Principal Component Analysis (PCA) and Recursive Feature Elimination (RFE) for feature extraction and optimization. The methodology focuses on refining facial feature representations to improve classification accuracy, which is critical for accurately detecting emotions like happiness, sadness, fear, and surprise. The approach was applied to two widely recognized facial emotion datasets: the Cohn-Kanade (CK) and the Japanese Female Facial Expression (JAFFE) datasets. By integrating PCA and RFE, the model efficiently selects the most relevant features, enhancing the overall performance of emotion recognition. A comparative analysis with existing deep learning models highlights the advantages of the proposed method. The effectiveness of this approach is further supported by the results, where the model achieves an accuracy of 98.49% on the CK dataset and 96.03% on the JAFFE dataset. These results demonstrate a significant improvement over recent methods, indicating the model's potential for real-world applications.

Research on work-stress recognition for deep ground miners based on depth-separable convolutional neural network

In deep mine production operations, the challenging operating environment intensifies the workload and pressure on coal miners. Long-term exposure to high-intensity operating pressure can seriously impact the physical and mental health of miners, leading to unsafe behaviors and accidents. To identify the pressure of miners' operations, this paper examines various driving scenarios, such as the deep-well tunneling machine cutting the wall and opening the alley, the shoveling machine shoveling ore, and the pickup truck driver transporting. The paper randomly collects facial images of miners during each operation using an explosion-proof CCD camera to obtain the facial expression characteristic data of miners. The Ferface2013 facial expression dataset was used to establish the dataset. The depth separable convolutional neural network MiniXception was used for training and to output the classification results of the pressure degree of deep shaft miners. A MiniXception-based miners' operating pressure recognition model was established. The training time, precision, recall, F1 score, and classification accuracy confusion matrix were selected. The study evaluated the effectiveness of the recognition model by measuring its training time, precision, recall, F1 score, and classification accuracy confusion matrix. The results indicate that the model has a correct recognition rate of 88% for the pressure state, 91% for the pleasure state, and 74% for the normal state. The overall accuracy of the model is 0.843. Therefore, the MiniXception recognition model is suitable for recognizing the pressure of miners' operations in deep mines. This can meet practical needs and is useful for preventing major accidents in mines, managing on-site safety, and managing safety in non-hazardous areas. It has important theoretical and practical significance.

Stress Detection through Compound Facial Expressions Using Neural Networks

Human face is the most dynamic part of the body that conveys information about the instant emotions. Facial expression analysis starts from early 1900s where later on scientists identify the six basic facial expressions as Anger, Disgust, Fear, Happiness, Sadness, Surprise and Neutral with the pioneering studies of psychologists. In the last decades, the acceleration in artificial intelligence and computer vision research makes it possible to automatically detect facial expressions through images. Furthermore, micro expressions, muscle movements and compound facial expressions; that are the combinations of the basic expressions can be also analyzed with computer vision algorithms. The main motivation in automatic facial expression analysis is to support human-computer. Furthermore, facial expression analysis can be a driver for automatic emotion analysis. In this study, we propose a novel method to detect stress indicators on the frontal face images. The detection procedure is based on compound facial expression analysis. 49 couples of 6 basic facial expressions where one is dominating, and the other is the complementary expression are employed. iCV-MEFED facial expression dataset is used in the experiments where video and image samples are provided for every compound facial expression class. The training and testing of compound facial expressions are done using a deep neural network. The robust representations of faces are achieved using a fusion method that combines deep texture features and the action units on the face. Then, through the appropriate grouping of the compound expressions, the system can detect the signs of stress. The proposed approach obtains encouraging results, and it is open to further improvements.

Facial Expression Analysis Using Multi-label Classification

Emotions form the reflection of an individual's mental state. Automatic expression analysis and its recognition based on their facial landmarks are of critical use, particularly to understand the emotional states of physically disabled individuals like autistic children, deaf, dumb, and bedridden individuals. This approach helps in interpreting human emotions, which is crucial for designing computational models and interfaces that cater to their needs. While traditional research has focused on six basic emotion categories (happiness, surprise, sadness, anger, fear, disgust), your work expands this by exploring compound emotion categories. Compound emotions combine these basic categories to create nuanced emotional states. The research utilizes the CFEE_Database_230 dataset of facial expressions captured for analysis and training purposes. The proposed methodology has the following three steps: Analyze the dataset and extract the region of interest (ROI), Extract various statistical and potentially discriminating features and apply a multi-label classification approach to categorize sets of emotions. This involves comparing the feature values across different emotion classes and assigning appropriate labels, particularly focusing on compound emotion categories. Also, the research employs different classifiers and metrics to evaluate the effectiveness of the model. After applying the classification methods, the results are analyzed using various metrics to assess the accuracy and effectiveness of emotion recognition based on facial landmarks. Based on the findings, the Binary Relevance Method yielded the best performance with Mean Average Precision and hamming Loss of 0.7227±0.0344 and 0.1906±0.0227, respectively. Overall, the work contributes to advancing automatic emotion recognition by considering a broader range of emotional categories beyond the traditional basics. This is particularly beneficial for populations such as the physically disabled and autistic children, where traditional communication methods might be limited or challenging.

Facial length and angle feature recognition for digital libraries.

With the continuous progress of technology, facial recognition technology is widely used in various scenarios as a mature biometric technology. However, the accuracy of facial feature recognition has become a major challenge. This study proposes a face length feature and angle feature recognition method for digital libraries, targeting the recognition of different facial features. Firstly, an in-depth study is conducted on the architecture of facial action networks based on attention mechanisms to provide more accurate and comprehensive facial features. Secondly, a network architecture based on length and angle features of facial expressions, the expression recognition network is explored to improve the recognition rate of different expressions. Finally, an end-to-end network framework based on attention mechanism for facial feature points is constructed to improve the accuracy and stability of facial feature recognition network. To verify the effectiveness of the proposed method, experiments were conducted using the facial expression dataset FER-2013. The experimental results showed that the average recognition rate for the seven common expressions was 97.28% to 99.97%. The highest recognition rate for happiness and surprise was 99.97%, while the relatively low recognition rate for anger, fear, and neutrality was 97.18%. The data has verified that the research method can effectively recognize and distinguish different facial expressions, with high accuracy and robustness. The recognition method based on attention mechanism for facial feature points has effectively optimized the recognition process of facial length and angle features, significantly improving the stability of facial expression recognition, especially in complex environments, providing reliable technical support for digital libraries and other fields. This study aims to promote the development of facial recognition technology in digital libraries, improve the service quality and user experience of digital libraries.

Benchmarking deep Facial Expression Recognition: An extensive protocol with balanced dataset in the wild

Facial expression recognition (FER) is crucial in enhancing human-computer interaction. While current FER methods, leveraging various open-source deep learning models and training techniques, have shown promising accuracy and generalizability, their efficacy often diminishes in real-world scenarios that are not extensively studied. Addressing this gap, we introduce a novel in-the-wild balanced testing facial expression dataset designed for cross-domain validation, called BTFER. We rigorously evaluated widely utilized networks and self-designed architectures, adhering to a standardized protocol. Additionally, we explored different configurations, including input resolutions, class balance management, and pre-trained strategies, to ascertain their impact on performance. Through comprehensive testing across three major FER datasets and our in-depth cross-validation, we have ranked these network architectures and formulated a series of practical guidelines for implementing deep learning-based FER solutions in real-life applications. This paper also delves into the ethical considerations, privacy concerns, and regulatory aspects relevant to the deployment of FER technologies in sectors such as marketing, education, entertainment, and healthcare, aiming to foster responsible and effective use. The BTFER dataset and the implementation code are available in Kaggle and Github, respectively.

FORMICARY SWARM OPTIMIZED DEEP CNN FOR FACIAL EMOTION RECOGNITION FROM HUMAN FACIAL EXPRESSIONS

Facial emotion recognition (FER) is a dominant research area that captures the biological facial features and matches the facial data with existing databases to analyze the individual’s emotional state. Numerous techniques have been formulated for attaining effective FER. However, the occlusions, different head positions, deformed faces, motion blur under unrestricted settings, and complicated backgrounds make it complex to analyze the facial images. In this paper, the formicary swarm optimization-based deep convolutional neural network (FSO-opt DCNN) model is utilized for facial emotion detection for which JAFFE and the RAVDESS facial expression datasets are used. DCNNs are proficient with built-in feature extraction strategies from images to map the various facial expressions to the corresponding emotional states adopted for effective FER. In addition, the intensity, directional, and edge patterns as well as the correlation features extracted utilizing the hybrid textual pattern, RESNET 101 and VGG 16-based correlation modules assist the DCNN to attain the informative feature from the high-resolution images. Further, the formicary swarm optimization (FSO) is incorporated that effectively tunes the DCNN to capture the complex relationships between the learned features that excel the FER capability. Evaluating the metrics, the face recognition model using the RAVDESS dataset achieves notable efficiencies during a training percentage (TP) of 90%, with values of 97.51%, 95.48%, 99.55%, 95.48%, 97.48%, 96.47%, and a minimum loss of 2.49%. Simultaneously, the emotion detection model demonstrates robust efficiencies with metric values of 96.75%, 98.49%, 95.01%, 98.49%, 96.72%, 97.59%, and a minimum loss of 3.25%. Finally, the obtained results reveal the efficacy of the FSO-opt DCNN, particularly in face recognition tasks, as it outperforms the existing models across various datasets, showcasing its versatility and potential in facial analysis applications.

Multi-threshold deep metric learning for facial expression recognition

Feature representations generated through triplet-based deep metric learning offer significant advantages for facial expression recognition (FER). Each threshold in triplet loss inherently shapes a distinct distribution of inter-class variations, leading to unique representations of expression features. Nonetheless, pinpointing the optimal threshold for triplet loss presents a formidable challenge, as the ideal threshold varies not only across different datasets but also among classes within the same dataset. In this paper, we propose a novel multi-threshold deep metric learning approach that bypasses the complex process of threshold validation and markedly improves the effectiveness in creating expression feature representations. Instead of choosing a single optimal threshold from a valid range, we comprehensively sample thresholds throughout this range, which ensures that the representation characteristics exhibited by the thresholds within this spectrum are fully captured and utilized for enhancing FER. Specifically, we segment the embedding layer of the deep metric learning network into multiple slices, with each slice representing a specific threshold sample. We subsequently train these embedding slices in an end-to-end fashion, applying triplet loss at its associated threshold to each slice, which results in a collection of unique expression features corresponding to each embedding slice. Moreover, we identify the issue that the traditional triplet loss may struggle to converge when employing the widely-used Batch Hard strategy for mining informative triplets, and introduce a novel loss termed dual triplet loss to address it. Extensive evaluations demonstrate the superior performance of the proposed approach on both posed and spontaneous facial expression datasets.

Auto Diagnosis of Parkinson's Disease Via a Deep Learning Model Based on Mixed Emotional Facial Expressions.

Parkinson's disease (PD) is a common degenerative disease of the nervous system in the elderly. The early diagnosis of PD is very important for potential patients to receive prompt treatment and avoid the aggravation of the disease. Recent studies have found that PD patients always suffer from emotional expression disorder, thus forming the characteristics of "masked faces". Based on this, we thus propose an auto PD diagnosis method based on mixed emotional facial expressions in the paper. Specifically, the proposed method is cast into four steps: Firstly, we synthesize virtual face images containing six basic expressions (i.e., anger, disgust, fear, happiness, sadness, and surprise) via generative adversarial learning, in order to approximate the premorbid expressions of PD patients; Secondly, we design an effective screening scheme to assess the quality of the above synthesized facial expression images and then shortlist the high-quality ones; Thirdly, we train a deep feature extractor accompanied with a facial expression classifier based on the mixture of the original facial expression images of the PD patients, the high-quality synthesized facial expression images of PD patients, and the normal facial expression images from other public face datasets; Finally, with the well-trained deep feature extractor, we thus adopt it to extract the latent expression features for six facial expression images of a potential PD patient to conduct PD/non-PD prediction. To show real-world impacts, we also collected a new facial expression dataset of PD patients in collaboration with a hospital. Extensive experiments are conducted to validate the effectiveness of the proposed method for PD diagnosis and facial expression recognition.

Electrocardiogram-based Emotion Recognition using Convolutional Neural Networks

Introduction: Electrocardiogram-based facial emotion identification has been a widely spread field in the last few decades. Due to its non-linearity, non-stationary and noisy properties, it is a very difficult job to create a framework that is capable of recognizing emotions with a high recognition rate. background: Electrocardiogram based facial emotion identification has been widely spread field in last some decades. Due to its non-linearity, non-stationary and noisy properties, it is very difficult job to create a framework that is capable of recognizing emotions with high recognition rate. Method: In this work, we introduce a new framework for facial emotion detection based on feature creation using a topographic representation of ECG signal properties. The feature map is created using deep learning techniques, and further, extricated features are then used for classification techniques to detect facial emotion recognition. Result: The recognition results are achieved on two publicly available facial expression datasets, i.e., Ascertain and Dreamer. We illustrated the usefulness of our framework by comparing results with other existing methods. Conclusion: The recognition results prove that the introduced framework can enhance the identifying rate on various given datasets.

Lightweight facial expression estimation for mobile computing in portable device

AbstractFacial expression recognition has been studied for many years, especially with the development of deep learning. However, the existing researches still have the following two issues. Firstly, the intensity of facial expression is neglected. Secondly, the deep learning based approaches cannot be directly deployed in the devices with limited resources. In order to tackle these two issues, this paper proposes a lightweight facial expression estimation method using a shallow ordinal regression algorithm, which is deployed in a portable smart device for mobile computing in IoTs. Compared with classification based facial expression recognition methods, ordinal regression considers the intensity of facial expression to achieve better mean absolute error (MAE), which is validated by experiments on several public facial expression datasets. The simulation in portable device also demonstrates its effectiveness for mobile computing.

CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition

Occlusion or posture change of the face in natural scenes has typical asymmetry; however, an asymmetric face plays a key part in the lack of information available for facial expression recognition. To solve the problem of low accuracy of asymmetric facial expression recognition, this paper proposes a fusion of channel global features and a spatial local information expression recognition network called the “Channel–Spatial Integration Network” (CSINet). First, to extract the underlying detail information and deepen the network, the attention residual module with a redundant information filtering function is designed, and the backbone feature-extraction network is constituted by module stacking. Second, considering the loss of information in the local key area of face occlusion, the channel–spatial fusion structure is constructed, and the channel features and spatial features are combined to enhance the accuracy of occluded facial recognition. Finally, before the full connection layer, more local spatial information is embedded into the global channel information to capture the relationship between different channel–spatial targets, which improves the accuracy of feature expression. Experimental results on the natural scene facial expression data sets RAF-DB and FERPlus show that the recognition accuracies of the modeling approach proposed in this paper are 89.67% and 90.83%, which are 13.24% and 11.52% higher than that of the baseline network ResNet50, respectively. Compared with the latest facial expression recognition methods such as CVT, PACVT, etc., the method in this paper obtains better evaluation results of masked facial expression recognition, which provides certain theoretical and technical references for daily facial emotion analysis and human–computer interaction applications.

A study on expression recognition based on improved mobilenetV2 network

This paper proposes an improved strategy for the MobileNetV2 neural network(I-MobileNetV2) in response to problems such as large parameter quantities in existing deep convolutional neural networks and the shortcomings of the lightweight neural network MobileNetV2 such as easy loss of feature information, poor real-time performance, and low accuracy rate in facial emotion recognition tasks. The network inherits the characteristics of MobilenetV2 depthwise separated convolution, signifying a reduction in computational load while maintaining a lightweight profile. It utilizes a reverse fusion mechanism to retain negative features, which makes the information less likely to be lost. The SELU activation function is used to replace the RELU6 activation function to avoid gradient vanishing. Meanwhile, to improve the feature recognition capability, the channel attention mechanism (Squeeze-and-Excitation Networks (SE-Net)) is integrated into the MobilenetV2 network. Experiments conducted on the facial expression datasets FER2013 and CK + showed that the proposed network model achieved facial expression recognition accuracies of 68.62% and 95.96%, improving upon the MobileNetV2 model by 0.72% and 6.14% respectively, and the parameter count decreased by 83.8%. These results empirically verify the effectiveness of the improvements made to the network model.

An algorithm for decomposing variations of 3D model

In recent times, there has been an increasing number of people who are concerned about the virtual reality field. Parameterization of deformations of 3D models is a meaningful problem in theoretical research and application development of virtual reality. This paper proposes a technique for conditional decomposition of 3D model variations based on a given set of 3D observations of an object, along with a set of input strain weights. The proposed algorithm is conducted through an optimal iterative process with solving the non-negative least squares problem. The output of the technique is a set of base models corresponding to different types of strain. The result of the proposed technique allows the creation of a new 3D model variant of the object in a simple and visually observable way. The algorithm has been tested and proven effective on data that are 3D face models created from the Japanese Female Facial Expression (JAFFE) dataset with labeled expression weights.

An evolutionary and insightful graph neural network based hybrid model for deciphering tenacious stress detection of humans using facial emotion recognition: coinage to affective computing

In the realm of affective computing, facial emotion recognition plays a pivotal role in ensuring prooper stres detection in humans. Within the domain of facial emotion recognition, accurate detection of emotions from facial expressions holds immense significance. Existing methods often neglect intricate connections between various facial features, potentially leading to challenges in discerning naunced emotional states. In response to this, we introdue approach termed Graph Neural Network-based Facial Emotion Recognition. Our technique harnesses the capabilities of graph neural network (GNNs) to surmount these limitations. The process inititaes with the creation of a graph that encapsulates the similarity between individual facial expression samples. Subsequently, this graph is fed into the model for intricate feature mapping. The outputs generated by this integrated model ingeniously merge the feature insights from adjacent facial expression samples, thereby enhancing the representation of data for downstream emotion recognition tasks. The graph neural network model processed facial expression samples are then directed to an emotion adjacency matrix that discerns various emotional states. The final outcomes are the culmination of results from this classification process. By accurately defining the top samples exhibiting the most pronounced emotional cues, our method efficiency determines the prevailing emotional state.Through comprehensive experimentation involving generation of 99.89% accuracy on publicily available facial expression dataset, our results highlight this algorithms superiority. This approach showcases a substantial increase in recognition accuracy, outperforming the closest competing algorithms by a significant margin. This demonstrates the efficacy and promise of this algorithm in revolutionizing stress detection, potentially ushering in a new era of precise and nuanced stress analysis form facial expressions.

Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition

Illumination variation has been a long-term challenge in real-world facial expression recognition (FER). Under uncontrolled or non-visible light conditions, near-infrared (NIR) can provide a simple and alternative solution to obtain high-quality images and supplement the geometric and texture details that are missing in the visible (VIS) domain. Due to the lack of large-scale NIR facial expression datasets, directly extending VIS FER methods to the NIR spectrum may be ineffective. Additionally, previous heterogeneous image synthesis methods are restricted by low controllability without prior task knowledge. To tackle these issues, we present the first approach, called for NIR-FER Stochastic Differential Equations (NFER-SDE), that transforms face expression appearance between heterogeneous modalities to the overfitting problem on small-scale NIR data. NFER-SDE can take the whole VIS source image as input and, together with domain-specific knowledge, guide the preservation of modality-invariant information in the high-frequency content of the image. Extensive experiments and ablation studies show that NFER-SDE significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

Hypergraph-Guided Disentangled Spectrum Transformer Networks for Near-Infrared Facial Expression Recognition

With the strong robusticity on illumination variations, near-infrared (NIR) can be an effective and essential complement to visible (VIS) facial expression recognition in low lighting or complete darkness conditions. However, facial expression recognition (FER) from NIR images presents a more challenging problem than traditional FER due to the limitations imposed by the data scale and the difficulty of extracting discriminative features from incomplete visible lighting contents. In this paper, we give the first attempt at deep NIR facial expression recognition and propose a novel method called near-infrared facial expression transformer (NFER-Former). Specifically, to make full use of the abundant label information in the field of VIS, we introduce a Self-Attention Orthogonal Decomposition mechanism that disentangles the expression information and spectrum information from the input image, so that the expression features can be extracted without the interference of spectrum variation. We also propose a Hypergraph-Guided Feature Embedding method that models some key facial behaviors and learns the structure of the complex correlations between them, thereby alleviating the interference of inter-class similarity. Additionally, we construct a large NIR-VIS Facial Expression dataset that includes 360 subjects to better validate the efficiency of NFER-Former. Extensive experiments and ablation studies show that NFER-Former significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors’ Satisfaction

Conventional studies on the satisfaction of museum visitors focus on collecting information through surveys to provide a one-way service to visitors, and thus it is impossible to obtain feedback on the real-time satisfaction of visitors who are experiencing the museum exhibition program. In addition, museum practitioners lack research on automated ways to evaluate a produced content program's lifecycle and its appropriateness. To overcome these problems, we propose a novel multi-convolutional neural network, called VimoNet, which is able to recognize visitors emotions automatically in real-time based on their facial expressions and body gestures. Furthermore, we design a user preference model of content and a framework to obtain feedback on content improvement for providing personalized digital cultural heritage content to visitors. Specifically, we define seven emotions of visitors and build a dataset of visitor facial expressions and gestures with respect to the emotions. Using the dataset, we proceed with feature fusion of face and gesture images trained on the DenseNet-201 and VGG-16 models for generating a combined emotion recognition model. From the results of the experiment, VimoNet achieved a classification accuracy of 84.10%, providing 7.60% and 14.31% improvement, respectively, over a single face and body gesture-based method of emotion classification performance. It is thus possible to automatically capture the emotions of museum visitors via VimoNet, and we confirm its feasibility through a case study with respect to digital content of cultural heritage.