Facial Expression Classification Research Articles

Conditional Generative Adversarial Net based Feature Extraction along with Scalable Weakly Supervised Clustering for Facial Expression Classification

Article Conditional Generative Adversarial Net based Feature Extraction along with Scalable Weakly Supervised Clustering for Facial Expression Classification Ze Chen 1, Lu Zhang 2, Jiaming Tang 3, Jiafa Mao 3, and Weiguo Sheng 1,* 1 Department of Computer Science, Hangzhou Normal University, Hangzhou, P.R. China 2 China Telecom Hangzhou Branch, Hangzhou, P.R. China 3 School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, P.R. China * Correspondence: w.sheng@ieee.org Received: 28 September 2023 Accepted: 30 June 2024 Published: 24 December 2024 Abstract: Extracting proper features plays a pivotal role in facial expression recognition. In this paper, we propose to extract facial expression features via a conditional generative adversarial net, followed by an algorithmic optimization step. These refined features are subsequently integrated into a scalable weakly supervised clustering framework for facial expression classification. Our results show that the proposed method can achieve an average recognition rate of 85.3%, which significantly outperforms related methods. Further, by employing a residual-based scheme for feature extraction, our method shows superior adaptability compared to algorithms based solely on weakly supervised clustering. Additionally, our method does not require high accurate annotation data and is robust to the noise presented in data sets.

Integration of Discriminative Information from Expressive and Neutral Face Image for Effective Modelling of Facial Expression Classification Problem

Integration of Discriminative Information from Expressive and Neutral Face Image for Effective Modelling of Facial Expression Classification Problem

Retraction Note: A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted boltzmann machine (RBM) with preprocessing for facial expression classification

Retraction Note: A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted boltzmann machine (RBM) with preprocessing for facial expression classification

Facial Expression Recognition-You Only Look Once-Neighborhood Coordinate Attention Mamba: Facial Expression Detection and Classification Based on Neighbor and Coordinates Attention Mechanism.

In studying the joint object detection and classification problem for facial expression recognition (FER) deploying the YOLOX framework, we introduce a novel feature extractor, called neighborhood coordinate attention Mamba (NCAMamba) to substitute for the original feature extractor in the Feature Pyramid Network (FPN). NCAMamba combines the background information reduction capabilities of Mamba, the local neighborhood relationship understanding of neighborhood attention, and the directional relationship understanding of coordinate attention. The resulting FER-YOLO-NCAMamba model, when applied to two unaligned FER benchmark datasets, RAF-DB and SFEW, obtains significantly improved mean average precision (mAP) scores when compared with those obtained by other state-of-the-art methods. Moreover, in ablation studies, it is found that the NCA module is relatively more important than the Visual State Space (VSS), a version of using Mamba for image processing, and in visualization studies using the grad-CAM method, it reveals that regions around the nose tip are critical to recognizing the expression; if it is too large, it may lead to erroneous prediction, while a small focused region would lead to correct recognition; this may explain why FER of unaligned faces is such a challenging problem.

Valence sound symbolism facilitates classification of vowels and emotional facial expressions.

Research on iconicity or sound symbolism chronicles associations between aspects of word meaning and word form. In valence sound symbolism, specifically, vowels were demonstrated to be associated with valence, so that the vowel /i/ (as in English sea) is judged to fit positively valenced stimuli, whereas the vowel /o/ (as in German Boot) is judged to fit negatively valenced stimuli. However, because previous findings were based on judgments or name invention tasks, they may reflect deliberate strategies. The present research aims to demonstrate the automaticity of valence sound symbolism in online language processing by means of a speeded response task. A total of 163 participants performed a speeded categorization task in which they were requested to classify one of two simultaneously presented stimuli, pseudowords containing the vowels /i/ or /o/ and pictures with positive or negative emotional facial expressions. Thus, stimulus pairings could be either congruent (/i/-words and positive expressions; /o/-words and negative expressions) or incongruent (/i/-words and negative expressions; /o/-words and positive expressions). We observed faster and more accurate responses for congruent over incongruent combinations (the simple effects were significant for positive but not for negative expressions). Thus, sound symbolic congruency (vs. incongruency) of an irrelevant stimulus facilitated responses, which accords with an automatic process contributing to valence sound symbolism and indicates that valence sound symbolism is deeply engrained in the human cognitive system. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Improved Histogram of Oriented Gradient (HOG) Feature Extraction for Facial Expressions Classification

Facial expression classification system is one of the implementations of machine learning (ML) that takes facial expression datasets, undergoes training, and then utilizes the trained results to recognize facial expressions in new facial images. The recognized facial expressions include anger, contempt, disgust, fear, happy, sadness, and surprise expressions. The method employed for facial feature extraction utilizes histogram-oriented gradient (HOG). This study proposes an enhancement method for HOG feature extraction by reducing the feature dimension into multiple sub-features based on gradient orientation intervals, referred to as HOG channel (HOG-C). Classifier testing techniques are divided into two methods for comparison—support vector machines (SVM) with HOG features and SVM with HOG-C features. The testing results demonstrate that SVM with HOG achieves an accuracy of 99.9% with an average training time of 18.03 minutes, while SVM with HOG-C attains a 100% accuracy with an average training time of 18.09 minutes. The testing outcomes reveal that the implementation of SVM with HOG-C successfully enhances accuracy for facial expression classification.

Facial Expression Recognition and Classification for Autism Spectrum Disorder based on SDFT Images using Deep Learning

Facial Expression Recognition and Classification for Autism Spectrum Disorder based on SDFT Images using Deep Learning

A Real-Time and Privacy-Preserving Facial Expression Recognition System Using an AI-Powered Microcontroller

This study proposes an edge computing-based facial expression recognition system that is low cost, low power, and privacy preserving. It utilizes a minimally obtrusive cap-based system designed for the continuous and real-time monitoring of a user’s facial expressions. The proposed method focuses on detecting facial skin deformations accompanying changes in facial expressions. A multi-zone time-of-flight (ToF) depth sensor VL53L5CX, featuring an 8 × 8 depth image, is integrated into the front brim of the cap to measure the distance between the sensor and the user’s facial skin surface. The distance values corresponding to seven universal facial expressions (neutral, happy, disgust, anger, surprise, fear, and sad) are transmitted to a low-power STM32F476 microcontroller (MCU) as an edge device for data preprocessing and facial expression classification tasks utilizing an on-device pre-trained deep learning model. Performance evaluation of the system is conducted through experiments utilizing data collected from 20 subjects. Four deep learning algorithms, including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) networks, and Deep Neural Networks (DNN), are assessed. These algorithms demonstrate high accuracy, with CNN yielding the best result, achieving an accuracy of 89.20% at a frame rate of 15 frames per second (fps) and a maximum latency of 2 ms.

Role of contingency in reward–emotion interactions during the categorization of emotional facial expressions.

Role of contingency in reward–emotion interactions during the categorization of emotional facial expressions.

Currency Classification Using Deep Learning

Among the uses of machine learning is the recognition of facial expressions. Based on the features that are derived from an image, it assigns a facial expression to one of the classes of facial expressions. Convolutional Neural Network (CNN) is a classification technique that may also be used to identify patterns in an image. We used the CNN approach to identify facial expressions in our proposed study. To increase the precision of facial emotion recognition, the wavelet transform is used after CNN processing. Seven distinct facial expressions are included in the facial expression image dataset that was obtained from Kaggle. The findings of the face expression recognition experiment utilizing CNN and wavelet transform show that the accuracy is improved, and the output is audible. Index Terms: Facial and Convolutional Neural Nets

Optimizing Facial Expression Recognition with Image Augmentation Techniques: VGG19 Approach on FERC Dataset

In the field of facial expression recognition (FER), the availability of balanced and representative datasets is key to success in training accurate models. However, Facial Expression Recognition Challenge (FERC) datasets often face the challenge of class imbalance, where some facial expressions have a much smaller number of samples compared to others. This issue can result in biased and unsatisfactory model performance, especially in recognizing less common facial expressions. Data augmentation techniques are becoming an important strategy as they can expand the dataset by creating new variations of existing samples, thus increasing the variety and diversity of the data. Data augmentation can be used to increase the number of samples for less common facial expression classes, thus improving the model's ability to recognize and understand diverse facial expressions. The augmentation results are then combined with balancing techniques such as SMOTE coupled with undersampling to improve model performance. In this study, VGG19 is used to support better model performance. This will provide valuable guidelines for optimizing more advanced CNN models in the future and may encourage further research in creating more innovative augmentation techniques.

Improving Facial Expression Classification through Ensemble Deep Learning Models

Improving Facial Expression Classification through Ensemble Deep Learning Models

Interoceptive sensitivity and perception of others' emotions: an investigation based on a two-stage model.

Recent research shows that sensitivity to interoceptive sensitivity is associated with a more granular experience of emotions. These studies suggest that individuals sensitive to their interoceptive signals can better perceive somatic physiological changes as compared to their counterparts. Therefore, they discriminate among a wide and subtle range of emotions. Further, the perception of others' emotions could be based on our own emotional experiences. However, whether interoceptive sensitivity is related to the perception of others' emotions remains unclear. Therefore, this study examined the relationship between interoceptive sensitivity and emotional perception. Considering the model that emotion perception comprises two processes, categorization of facial expressions and approach-avoidance responses, this study examined both categorizations of facial expressions and approach-avoidance responses. The results showed no relationship between interoceptive sensitivity and the perception of emotion, which suggests that interoceptive sensitivity is related to the experience of emotion but does not affect the granularity of emotional perception. Future studies should diversely and empirically examine the role of the body in emotional perception from the perspective of interoceptive sensitivity.

Enhancing image-based facial expression recognition through muscle activation-based facial feature extraction

This article introduces a non-intrusive method to estimate facial muscle activity from images, diverging from conventional electrode-based approaches. Our methodology capitalizes on an inclusive set of features encompassing a diverse range of facial muscles, often overlooked in research, thus significantly expanding the scope of analyzing muscle activity within facial expressions. Our method is based on the standard 68-point face landmark and extends it by identifying the interactions of these muscles when a person performs a specific facial expression. These interactions are recorded in a feature vector, which is used by three classifiers, Linear Discriminant (LD), Support Vector Machine (SVM) and Multi-layer Perceptron (MLP) to classify six facial expressions (anger, disgust, fear, happiness, neutrality, and sadness). The method’s validation is conducted with three databases: FACES, KDEF, and JAFFE. These databases present different challenges; the first contains faces of individuals from three age ranges and both genders displaying facial expressions. On one hand, the KDEF database contains images of both genders but only within the range of young adults, whereas JAFFE comprises solely female faces. Additionally, JAFFE presents another drawback, representing only 10% of the images contained in FACES. In all these cases, our method yields excellent results, occasionally achieving 100% classification, especially in the young category. It is also noteworthy that the best results were obtained in the classification of facial expressions in female faces, suggesting that women tend to be more expressive. The method was thoroughly evaluated, and the results demonstrate the robustness of the approach, showcasing its good performance across three different databases and also for faces of individuals across various age groups. Numerical metrics, including accuracy, precision, recall, and F1 score, are presented along with confusion matrices.

Emotion-driven action interaction design in small quadruped robots: leveraging NXEIK multilevel network

This study introduces a facial expression-driven interaction method for small quadruped robots, featuring three layers: Perception (L1), Analysis (L2), and Expression (L3). L1 focuses on data acquisition and image stabilization, L2 on model training and emotion classification, and L3 on control feedback and interactive movements. The core NXEIK network model comprises the NAFNet for image stabilization, Mini-Xception for facial recognition, EANet for action mapping, and an inverse kinematics model. Validation on a self-designed robot platform demonstrated the method's ability to enhance jittery image data and outperform deep learning networks like ViT and ResNet-101 in facial expression classification. The NXEIK model enables the robot to adapt its movements to various expressions using only three parameter types. This research provides a feasible solution for enhancing human-robot interaction and movement design for flexible quadruped robots through facial expressions.

Supplemental Material for Role of Contingency in Reward–Emotion Interactions During the Categorization of Emotional Facial Expressions

Supplemental Material for Role of Contingency in Reward–Emotion Interactions During the Categorization of Emotional Facial Expressions

Completed Robust Local Binary Pattern Texture Descriptor for Classification of Facial Expression

Completed Robust Local Binary Pattern Texture Descriptor for Classification of Facial Expression

AI-based cancer pain assessment through speech emotion recognition and video facial expressions classification

AI-based cancer pain assessment through speech emotion recognition and video facial expressions classification

Application of Multi-sensory interaction design Based on Machine Learning in Virtual Reality

Virtual reality (VR) technology is well-known to the public for its unparalleled sense of environmental immersion, which is accompanied by a large amount of human-computer interaction between users and objects in the virtual environment. Users transmit and obtain information through their senses, so the processing mechanism of information in various senses under the background of VR technology has become the foundation of research. In the process of natural information dissemination, users’ senses are synergistically involved, so whether there is a correlation between various senses that affects users’ final perception has become an important theoretical support for studying multi-sensory interaction. Facial expression recognition is an important component of multi-sensory interaction technology. Therefore, this article mainly uses machine learning algorithms to achieve real-time detection of facial targets in images or videos, as well as classification and recognition of facial expressions. To meet the task requirements of the expression recognition module, this article selects the DeepID network model based on deep learning in the field of facial recognition as the basic model for the expression recognition function. This study utilizes the excellent feature extraction ability of the original network for facial images and introduces residual modules and attention mechanisms, making the modified network model more suitable for facial expression multi classification recognition tasks. The recognition accuracy of the facial expression classification algorithm proposed by this work reaches 97.2%.

Comparative Study of K-NN, Naive Bayes and SVM for Face Expression Classification Techniques

Abstract Classification is a data mining technique used to predict group membership for data instances within a given dataset. Classification is considered as an example of supervised learning as training data associated with class labels is given as input. This comparative study focuses on the application of K-Nearest Neighbors (K-NN), Naive Bayes, and Support Vector Machines (SVM) classification techniques in the context of face expression recognition. The study aims to evaluate and compare the performance of these techniques in accurately classifying facial expressions, their advantages and disadvantages. Facial expression recognition is an important area in computer vision and emotion analysis. It involves identifying and classifying different facial expressions such as happiness, sadness, anger, fear, surprise and disgust. Accurate classification of facial expressions is crucial for various applications, including human-computer interaction, affective computing and psychological research. K-NN is a non-parametric algorithm that classifies data based on the majority class of its K-nearest neighbors. Naive Bayes is a probabilistic algorithm that makes classification decisions based on the Bayes theorem and assumes independence between features. SVM is a machine learning algorithm that constructs a hyperplane to separate different classes of data. Dataset such as Affectnet is used to conduct the study including labeled facial expression images. Three algorithms are tested on this dataset to evaluate their performance in correctly classifying facial expressions. Performance metrics such as accuracy, precision, recall, and F1-score are used to assess the classification performance of each technique. Software used for conducting the experiments is Python.