Approach For Facial Expression Recognition Research Articles

Facial emotion recognition based on upper features and transfer learning

Facial expression recognition (FER) in the upper face focuses on the analysis and recognition of emotions based on features extracted from the upper region of the face. This region typically affects the eyes, eyebrows, forehead, and sometimes the upper cheeks. Since these areas are often less affected by face masks or other facial coverings, FER algorithms can concentrate on capturing and interpreting the relevant facial cues, such as eye movements, eyebrow positions, and forehead wrinkles, to accurately recognize and classify different emotions. By focusing on the upper face, FER systems can mitigate the impact of occlusions caused by masks and still provide meaningful insights into the emotional states of individuals. In this work, a FER approach focusing on the upper region is proposed. Several experiments have been made using the CK+ dataset in addition to a comparison between the emotion recognition scores using the upper and the entire face in order to determine whether this area can reflect the real expressed emotion. The results of our approach are promising compared to previous studies with an accuracy up to 96%.

Facial Biosignals Time–Series Dataset (FBioT): A Visual–Temporal Facial Expression Recognition (VT-FER) Approach

Visual biosignals can be used to analyze human behavioral activities and serve as a primary resource for Facial Expression Recognition (FER). FER computational systems face significant challenges, arising from both spatial and temporal effects. Spatial challenges include deformations or occlusions of facial geometry, while temporal challenges involve discontinuities in motion observation due to high variability in poses and dynamic conditions such as rotation and translation. To enhance the analytical precision and validation reliability of FER systems, several datasets have been proposed. However, most of these datasets focus primarily on spatial characteristics, rely on static images, or consist of short videos captured in highly controlled environments. These constraints significantly reduce the applicability of such systems in real-world scenarios. This paper proposes the Facial Biosignals Time–Series Dataset (FBioT), a novel dataset providing temporal descriptors and features extracted from common videos recorded in uncontrolled environments. To automate dataset construction, we propose Visual–Temporal Facial Expression Recognition (VT-FER), a method that stabilizes temporal effects using normalized measurements based on the principles of the Facial Action Coding System (FACS) and generates signature patterns of expression movements for correlation with real-world temporal events. To demonstrate feasibility, we applied the method to create a pilot version of the FBioT dataset. This pilot resulted in approximately 10,000 s of public videos captured under real-world facial motion conditions, from which we extracted 22 direct and virtual metrics representing facial muscle deformations. During this process, we preliminarily labeled and qualified 3046 temporal events representing two emotion classes. As a proof of concept, these emotion classes were used as input for training neural networks, with results summarized in this paper and available in an open-source online repository.

Multi-head Network based Students Behaviour Prediction with Feedback Generation for Enhancing Classroom Engagement and Teaching Effectiveness

Emotions are pivotal in the learning process, highlighting the importance of identifying students' emotional states within educational settings. While neural network models, particularly those rooted in deep learning, have demonstrated remarkable accuracy in detecting primary emotions like happiness, sadness, fear, disgust, and anger from facial expressions in videos, these emotions occur infrequently in learning environments. Conversely, cognitive emotions such as engagement, confusion, frustration, and boredom are significantly more prevalent, transpiring five times more frequently than basic emotions. However, unlike basic emotions which are relatively distinct, cognitive emotions present a subtler distinction, necessitating the utilization of more sophisticated models for accurate recognition. The proposed work presents an efficient Facial Expression Recognition (FER) model for monitoring the student engagement in a learning environment by considering their facial expressions like boredom, frustration, confusion and engagement. The proposed methodology includes certain pre-processing steps followed by facial expression recognition founded on Efficient-Net B3 CNN in which the learning parameters are optimized using Circle-Inspired Optimization Algorithm (CIOA). Finally, the post processing stage estimates the frame-wise group engagement level (GEL) of students based on certain expression labels. Based on the acquired results, it is noted that the suggested Efficient-Net B3 CNN-CIOA based FER model provides promising results in terms of accuracy by 99.5%, precision by 99.2%, recall by 99.5% and f1-score by 99.6%, when compared with some state-of-art facial expression recognition approaches. Also, the suggested approach computational complexity is very much less than the compared existing approaches.

HSAW: A Half-Face Self-Attention Weighted Approach for Facial Expression Recognition

Facial expression recognition plays an increasingly important role in daily life, and it is used in several areas of human–computer interaction, such as robotics, assisted driving, and intelligent tutoring systems. However, the current mainstream methods are based on the whole face, and do not consider the existence of expression asymmetry between the left and right half-face. Hence, the accuracy of facial expression recognition needs to be improved. In this paper, we propose a half-face self-attention weighted approach called HSAW. Using statistical analysis and computer vision techniques, we found that the left half-face contains richer expression features than the right half-face. Specifically, we employed a self-attention mechanism to assign different weights to the left and right halves of the face. These weights are combined with convolutional neural network features for improved facial expression recognition. Furthermore, to attack the presence of uncertain categories in the dataset, we introduce adaptive re-labeling module, which can improve the recognition accuracy. Extensive experiments conducted on the FER2013 and RAF datasets have verified the effectiveness of the proposed method, which utilizes fewer parameters.

CL-TransFER: Collaborative learning based transformer for facial expression recognition with masked reconstruction

Facial expression recognition (FER) has attracted intensive attention due to its critical role in various computer vision tasks. However, existing FER approaches suffer from either noisy annotations or expression ambiguity (high inter-class and low intra-class similarity), limiting the FER performance. To this end, we propose a robust end-to-end collaborative learning based transformer for FER (CL-TransFER) in this paper. Specifically, CL-TransFER co-trains a CNN feature extractor and a transformer feature extractor jointly to extract both rich local semantic features as well as global structural information from facial images. By enforcing the consensus between the predictions of two extractors, the CL-TransFER could suppress the influence of noisy annotations. To further tackle the expression ambiguity problem, we design a simple yet efficient self-supervised masked reconstruction (SSMR) task to pre-train the transformer feature extractor of CL-TransFER. This enhances the model's capability of learning fine-grained discriminative representations. Extensive experiments on three popular benchmarks have demonstrated the effectiveness and superiority of our method.

Geometrical Facial Expression Recognition Approach Based on Fusion CNN-SVM

Geometrical Facial Expression Recognition Approach Based on Fusion CNN-SVM

A novel PSO-ViT approach for facial emotion recognition

ABSTRACT Facial Emotion Recognition is gaining interest from researchers in various fields due to its numerous applications. The Vision Transformer (ViT) outperforms convolutional neural network (CNN)-based systems significantly in terms of performance when compared to contemporary image categorisation algorithms. In this context, the primary objective of this paper is to present an innovative and efficient deep learning model for facial emotion recognition. This hybrid intelligent model combines the Vision Transformer with Particle Swarm Optimization (PSO). The proposed model accurately identified the majority of emotions in images, achieving an accuracy of 95.93% for the Fer2013plus dataset and 100% for the CK+ dataset, as demonstrated through experiments conducted on both datasets. According to the results, the proposed facial expression recognition approach has either outperformed or closely matched the performance of state-of-the-art methods.

Dataset classification: An efficient feature extraction approach for grammatical facial expression recognition

In this paper, an efficient features extraction using validated statistical approaches is proposed, along with a robust Grammatical Facial Expressions (GFEs) classifier in facial expression recognition systems. Accordingly, a new dataset was collected from 70 participants (33 males and 37 females) ranging in age from 18 to 46. The total number of video clips collected was 765.The features extracted in this study consist of 17 features associated with three categories of non-manual features: facial expression, head movement, and eye-gaze. Automatic recognition of nine classes of grammatical facial expressions in two languages (Arabic and Persian) is performed using a linear Support Vector Machine (SVM) classifier.The proposed system was also validated by testing it on the American Sign Language (ASL) dataset. In comparison to previous works on the ASL dataset, the results showed a higher accuracy rate of 95%.

Designing a Dual-level Facial Expression Evaluation System for Performers Using Geometric Features and Petri Nets

The existing methods of Facial Expression Recognition (FER) primarily analyze six basic expressions, namely, surprise, happiness, anger, sadness, fear, and disgust. The Indian performing arts use three more well-defined expressions—peaceful, proud, and erotic. This study proposes an intelligent dual-level expression evaluation system that classifies performance-specific expressions into nine classes, assigns intensity level to the expression, and suggests modifications to the user for precise exhibition of an expression. At decision level-1 of a dual-level system, an 11-state model is designed to classify the nine expressions. The model is verified using the Colored Petri Net that helps analyze the rules used for the classification. Decision level-1 is also implemented using input feature database and SVM classifier, which yields 95.77% accuracy. Further, at decision level-2, SVM is used to assign an intensity level to the correctly classified images. In case of incorrectly exhibited expressions, feedback is provided to the user about the incorrect facial component state. The application-specific image dataset is used for the present study. The qualitative comparison with the other FER approaches is also carried out. With the increasing popularity of Indian classical dance in Western and Asian countries, the dual-level system enables learners of performing arts to practice, evaluate, and improvise their expression skills.

Deep Learning Approach for Facial Expression Recognition

Abstract: In recent decades, facial expression recognition has emerged as a hot topic with significant implications in the realm of human-computer interaction. The simplest way for humans to express their emotions is through facial expressions. Non-verbal communication relies heavily on facial expression. This study outlines deep learning-based Facial Expression Recognition (FER) algorithm. The performance of the FER approach is compared based on the number of expressions detected and the difficulty of CNN algorithms. The FER2013, CK+ databases were used in testing the design. CNNs (Convolutional Neural Networks) have as of late gained fame in the field of profound learning because of its superb plan and capacity to convey clever outcomes without the requirement for manual feature extraction from raw information. The suggested algorithm achieves a higher rate of recognition on four datasets.

Spatio-temporal modelling with multi-gradient features and elongated quinary pattern descriptor for dynamic facial expression recognition

We propose a new spatio-temporal modelling approach for Dynamic Facial Expression Recognition (DFER). We first convert the domain of the spatial images in the sequence to the gradient of magnitude and angle images at different orientations. Robust gradient components are developed to deal with difficult types of illuminations, such as darkness, by forming the eight edge responses of the Gaussian mask. To describe the dynamic Facial Expression (FE) changes we extend the Elongated Quinary Pattern (EQP) descriptor to encode separately the anisotropic structure of the uniform patterns from Three Orthogonal Planes (TOP) of each gradient sequence. Then each encoded sequence is divided into a stack of block volumes in the XY, XT and YT planes. For each plane, the co-occurrence of histogram features are calculated from each block volume and concatenated together. Simple three-dimensional histogram features are generated by concatenating the histogram features of all planes. A Multi Classifier System (MCS) based on a multi-class Support Vector Machine (SVM) is adopted to combine all scores for the encoded sequences. The proposed approach is evaluated with the challenging MMI and Oulu-CASIA databases with different set-ups and advantage has been shown in terms of generalisation to different databases, together with robustness against difficult pose variations and illumination changes. In terms of Recognition Accuracy (RA), a comparison is established with DFER methods in the literature. A high recognition rate of 79.23% is attained in the case of six classes when applied to the MMI database which surpasses all the state-of-the-art results.

SCL-FExR: supervised contrastive learning approach for facial expression Recognition

SCL-FExR: supervised contrastive learning approach for facial expression Recognition

Facial Expression Recognition Robust to Occlusion and to Intra-Similarity Problem Using Relevant Subsampling

This paper proposes facial expression recognition (FER) with the wild data set. In particular, this paper chiefly deals with two issues, occlusion and intra-similarity problems. The attention mechanism enables one to use the most relevant areas of facial images for specific expressions, and the triplet loss function solves the intra-similarity problem that sometimes fails to aggregate the same expression from different faces and vice versa. The proposed approach for the FER is robust to occlusion, and it uses a spatial transformer network (STN) with an attention mechanism to utilize specific facial region that dominantly contributes (or that is the most relevant) to particular facial expressions, e.g., anger, contempt, disgust, fear, joy, sadness, and surprise. In addition, the STN model is connected to the triplet loss function to improve the recognition rate which outperforms the existing approaches that employ cross-entropy or other approaches using only deep neural networks or classical methods. The triplet loss module alleviates limitations of the intra-similarity problem, leading to further improvement of the classification. Experimental results are provided to substantiate the proposed approach for FER, and the result outperforms the recognition rate in more practical cases, e.g., occlusion. The quantitative result provides FER results with more than 2.09% higher accuracy compared to the existing FER results in CK+ data sets and 0.48% higher than the accuracy of the results with the modified ResNet model in the FER2013 data set.

A feature boosted deep learning method for automatic facial expression recognition.

Automatic facial expression recognition (FER) plays a crucial role in human-computer based applications such as psychiatric treatment, classroom assessment, surveillance systems, and many others. However, automatic FER is challenging in real-time environment. The traditional methods used handcrafted methods for FER but mostly failed to produce superior results in the wild environment. In this regard, a deep learning-based FER approach with minimal parameters is proposed, which gives better results for lab-controlled and wild datasets. The method uses features boosting module with skip connections which help to focus on expression-specific features. The proposed approach is applied to FER-2013 (wild dataset), JAFFE (lab-controlled), and CK+ (lab-controlled) datasets which achieve accuracy of 70.21%, 96.16%, and 96.52%. The observed experimental results demonstrate that the proposed method outperforms the other related research concerning accuracy and time.

Hybrid Convolutional Neural Network and Long Short-Term Memory Approach for Facial Expression Recognition

Facial Expression Recognition (FER) has been an important field of research for several decades. Extraction of emotional characteristics is crucial to FERs, but is complex to process as they have significant intra-class variances. Facial characteristics have not been completely explored in static pictures. Previous studies used Convolution Neural Networks (CNNs) based on transfer learning and hyperparameter optimizations for static facial emotional recognitions. Particle Swarm Optimizations (PSOs) have also been used for tuning hyperparameters. However, these methods achieve about 92 percent in terms of accuracy. The existing algorithms have issues with FER accuracy and precision. Hence, the overall FER performance is degraded significantly. To address this issue, this work proposes a combination of CNNs and Long Short-Term Memories (LSTMs) called the HCNN-LSTMs (Hybrid CNNs and LSTMs) approach for FERs. The work is evaluated on the benchmark dataset, Facial Expression Recog Image Ver (FERC). Viola-Jones (VJ) algorithms recognize faces from preprocessed images followed by HCNN-LSTMs feature extractions and FER classifications. Further, the success rate of Deep Learning Techniques (DLTs) has increased with hyperparameter tunings like epochs, batch sizes, initial learning rates, regularization parameters, shuffling types, and momentum. This proposed work uses Improved Weight based Whale Optimization Algorithms (IWWOAs) to select near-optimal settings for these parameters using best fitness values. The experimental findings demonstrated that the proposed HCNN-LSTMs system outperforms the existing methods.

Recognition of human emotions in real time

This article emphasizes the importance and relevance of using neural networks to recognize human emotions in real time. He explores the possibilities of using convolutional neural networks for efficient analysis and classification of emotions based on facial expressions. The article describes the process of developing an application using JavaScript and TensorFlow and highlights potential optimization opportunities to improve the speed and accuracy of emotion recognition. The solution proposed in this paper offers a highly effective approach to emotion recognition, applicable in a variety of fields ranging from medicine and social research to marketing and entertainment. This highlights the importance of advanced technologies that improve human-computer interaction and reflects the importance of using innovative methods to improve the quality of human life. The convolutional neural network (CNN) algorithm used in this study uses a deep learning approach for facial expression recognition. The architecture of the CNN model includes several convolutional layers, each of which is responsible for identifying specific features of face images, such as edges, textures, and patterns. These layers are followed by pooling layers to reduce dimensionality, allowing the network to focus on the most meaningful features. In addition, the use of rectified linear units (ReLU) as activation functions improves the model’s ability to capture complex relationships in the data. In addition, the paper examines a deep learning facial expression recognition (FER) model based on convolutional neural networks, demonstrating its effectiveness in accurately identifying and classifying different emotional states based on facial features. The robust architecture of the FER model and the learning process allow it to distinguish the nuances of facial expressions, contributing to a more detailed understanding of human emotions. Consequently, such a system could be useful for medicine, social research, marketing, and entertainment, helping to analyze emotional responses and improve user interaction strategies.

Role of Zoning in Facial Expression Using Deep Learning

Facial expression is an unspoken message essential to collaboration and effective discourse. An inner emotional state of a human is expressed using facial expressions and is very effective for communication with actual emotions. Anger, happiness, sadness, contempt, surprise, fear, disgust, and neutral are eight common expressions of humans. Scientific community proposed several face emotion recognition techniques. However, due to fewer face landmarks and their intensity for deep learning models, performance improvement for facial expression recognition still needs to be improved for accurately predicting facial emotion recognition. This study proposes a zoning-based face expression recognition (ZFER) to locate more face landmarks to perceive deep face emotions indemnity through zoning. After face extraction, landmarks from the face, such as the eyes, eyebrows, nose, forehead, and mouth, are extracted. The second step is zoning each landmark into four regions and zone-based face landmarks are passed to the VGG-16 model to generate a feature map. Finally, the feature map is given as input to fully connected neural network (FCNN) to classify facial emotions into multiple classes. Various experiments are performed on facial expression recognition (FER) 2013 and CK+ datasets to evaluate our proposed model with state-of-the-art facial expression recognition approaches using performance assessment metrics like accuracy. The accuracy of the proposed method with face features on CK+ and FER2013 are 98.4% and 65%, respectively. The experimental zoning results improve from 98.47% to 98.74% on the CK+ dataset.

A Comparative Approach for Facial Expression Recognition in Higher Education Using Hybrid-Deep Learning from Students' Facial Images

A Comparative Approach for Facial Expression Recognition in Higher Education Using Hybrid-Deep Learning from Students' Facial Images

Fine-grained facial expression recognition via relational reasoning and hierarchical relation optimization

Facial expression recognition (FER), aiming to recognize the type of facial expressions, has achieved significant progress. However, most of the existing FER approaches ignore the influence of structure relations among image set and the semantic associations between labels. Recently, some studies turn to explore fine-grained FER which includes hierarchical label structure, but they merely explore the influence of hierarchical relations of labels. Inspired by this, in this paper, we propose a relational reasoning and hierarchical relation optimization network (R3HO-Net) that explores the above three kinds of relations simultaneously. Concretely, we first construct two sub-graphs, i.e., intra-image graph (IIG) and intra-label graph (ILG). Meanwhile, we propose an entropy-based relation adaptive initialization strategy to construct a heterogeneous inter-graph (HIG). Then fine-grained stream of R3HO-Net, including a relational update GCN module, updates the graphs simultaneously in an iterative way and outputs the mapping probabilities between heterogeneous node pairs to infer the final mapping results. Moreover, we also propose a hierarchical label optimization module and a hierarchical optimization loss to optimize fine-grained prediction results. Extensive experiments on serveral benchmarks demonstrate the superiority of the proposed approach.

A Comparative Study of Local Descriptors and Classifiers for Facial Expression Recognition

Facial Expression Recognition (FER) is a growing area of research due to its numerous applications in market research, video gaming, healthcare, security, e-learning, and robotics. One of the most common frameworks for recognizing facial expressions is by extracting facial features from an image and classifying them as one of several prototypic expressions. Despite the recent advances, it is still a challenging task to develop robust facial expression descriptors. This study aimed to analyze the performances of various local descriptors and classifiers in the FER problem. Several experiments were conducted under different settings, such as varied extraction parameters, different numbers of expressions, and two datasets, to discover the best combinations of local descriptors and classifiers. Of all the considered descriptors, HOG (Histogram of Oriented Gradients) and ALDP (Angled Local Directional Patterns) were some of the most promising, while SVM (Support Vector Machines) and MLP (Multi-Layer Perceptron) were the best among the considered classifiers. The results obtained signify that conventional FER approaches are still comparable to state-of-the-art methods based on deep learning.