Automatic Facial Expression Recognition Research Articles

Modelling appearance variations in expressive and neutral face image for automatic facial expression recognition

AbstractIn automatic facial expression recognition (AFER) systems, modelling the spatio‐temporal feature information in a specific manner, coalescing, and its effective utilization is challenging. The state‐of‐the‐art studies have examined integrating multiple features to enhance the recognition rate of AFER systems. However, the feature variations between expressive and neutral face images are not fully explored to identify the expression class. The proposed research presents an innovative approach to AFER by modelling appearance variations in both expressive and neutral face images. The prominent contributions of the work are developing a novel and hybrid feature space by integrating the discriminative feature distribution derived from expressive and neutral face images; preserving the highly discriminative latent feature distribution using autoencoders. Local binary pattern (LBP) and histogram of oriented gradients (HOG) are the feature descriptors employed to derive the discriminative texture and shape information, respectively. The component‐based approach is employed, wherein the features are derived from the salient facial regions instead of the whole face. The three‐stage stacked deep convolutional autoencoder (SDCA) and multi‐class support vector machine (MSVM) are employed to address dimensionality reduction and classification, respectively. The efficacy of the proposed model is substantiated by empirical findings, which establish its superiority in terms of accuracy in AFER tasks on widely recognized benchmark datasets.

Automatic facial expression recognition under partial occlusion based on motion reconstruction using a denoising autoencoder

Automatic facial expression recognition (FER) plays a valuable role in various fields, including health, road safety, and marketing, where providing feedback on the user’s condition is crucial. While significant progress has been made in controlled environments (such as frontal, unconcluded, and well-lit conditions), recognizing facial expressions in unconstrained environments (natural settings) remains challenging. The presence of occlusions poses a particular difficulty as they obscure parts of the facial information captured in the image. To address this issue, researchers have proposed different solutions, broadly categorized into two approaches: those focusing on visible regions of the face and those attempting to reconstruct hidden parts. Currently, most solutions rely on texture or geometry-based methods, with only a few utilizing motion-based approaches. However, incorporating motion appears to be particularly promising in adapting to occlusions due to its unique characteristics, such as close-range propagation and local coherence. In this paper, our focus lies on leveraging motion to overcome the challenges posed by occlusions in FER tasks.

A novel deep learning model to improve the recognition of students’ facial expressions in online learning environments

With the fast development of artificial intelligence and emerging technologies, automatic recognition of students’ facial expressions has received increased attention. Facial expressions are a kind of external manifestation of emotional states. It is important for teachers to assess students’ emotional states and adjust teaching activities accordingly. However, existing methods for automatic facial expression recognition have the limitations of low accuracy of recognition and poor feature extraction. To address the problem, this study proposed a novel deep learning model called DenseNetX-CBAM to improve facial expression recognition by utilizing a variant of densely connected convolutional networks (DenseNet) to reduce unnecessary parameters and strengthen the reuse of expression features between networks; moreover, convolutional block attention module (CBAM) was integrated to allow the networks to focus on important special regions and important channels when representing features. The proposed model was tested using 217 video clips of 33 students in an online course. The results demonstrated promising effects of the method in improving the accuracy of facial expression recognition, which can help teachers to accurately recognize students’ emotions and provide real-time adjustment in online learning environments.

Postoperative accurate pain assessment of children and artificial intelligence: A medical hypothesis and planned study.

Although the pediatric perioperative pain management has been improved in recent years, the valid and reliable pain assessment tool in perioperative period of children remains a challenging task. Pediatric perioperative pain management is intractable not only because children cannot express their emotions accurately and objectively due to their inability to describe physiological characteristics of feeling which are different from those of adults, but also because there is a lack of effective and specific assessment tool for children. In addition, exposure to repeated painful stimuli early in life is known to have short and long-term adverse sequelae. The short-term sequelae can induce a series of neurological, endocrine, cardiovascular system stress related to psychological trauma, while long-term sequelae may alter brain maturation process, which can lead to impair neurodevelopmental, behavioral, and cognitive function. Children's facial expressions largely reflect the degree of pain, which has led to the developing of a number of pain scoring tools that will help improve the quality of pain management in children if they are continually studied in depth. The artificial intelligence (AI) technology represented by machine learning has reached an unprecedented level in image processing of deep facial models through deep convolutional neural networks, which can effectively identify and systematically analyze various subtle features of children's facial expressions. Based on the construction of a large database of images of facial expressions in children with perioperative pain, this study proposes to develop and apply automatic facial pain expression recognition software using AI technology. The study aims to improve the postoperative pain management for pediatric population and the short-term and long-term quality of life for pediatric patients after operational event.

A Deep Quantum Convolutional Neural Network Based Facial Expression Recognition For Mental Health Analysis.

The purpose of this work is to analyze how new technologies can enhance clinical practice while also examining the physical traits of emotional expressiveness of face expression in a number of psychiatric illnesses. Hence, in this work, an automatic facial expression recognition system has been proposed that analyzes static, sequential, or video facial images from medical healthcare data to detect emotions in people's facial regions. The proposed method has been implemented in five steps. The first step is image preprocessing, where a facial region of interest has been segmented from the input image. The second component includes a classical deep feature representation and the quantum part that involves successive sets of quantum convolutional layers followed by random quantum variational circuits for feature learning. Here, the proposed system has attained a faster training approach using the proposed quantum convolutional neural network approach that takes [Formula: see text] time. In contrast, the classical convolutional neural network models have [Formula: see text] time. Additionally, some performance improvement techniques, such as image augmentation, fine-tuning, matrix normalization, and transfer learning methods, have been applied to the recognition system. Finally, the scores due to classical and quantum deep learning models are fused to improve the performance of the proposed method. Extensive experimentation with Karolinska-directed emotional faces (KDEF), Static Facial Expressions in the Wild (SFEW 2.0), and Facial Expression Recognition 2013 (FER-2013) benchmark databases and compared with other state-of-the-art methods that show the improvement of the proposed system.

The cross-race effect in automatic facial expression recognition violates measurement invariance.

Emotion has been a subject undergoing intensive research in psychology and cognitive neuroscience over several decades. Recently, more and more studies of emotion have adopted automatic rather than manual methods of facial emotion recognition to analyze images or videos of human faces. Compared to manual methods, these computer-vision-based, automatic methods can help objectively and rapidly analyze a large amount of data. These automatic methods have also been validated and believed to be accurate in their judgments. However, these automatic methods often rely on statistical learning models (e.g., deep neural networks), which are intrinsically inductive and thus suffer from problems of induction. Specifically, the models that were trained primarily on Western faces may not generalize well to accurately judge Eastern faces, which can then jeopardize the measurement invariance of emotions in cross-cultural studies. To demonstrate such a possibility, the present study carries out a cross-racial validation of two popular facial emotion recognition systems-FaceReader and DeepFace-using two Western and two Eastern face datasets. Although both systems could achieve overall high accuracies in the judgments of emotion category on the Western datasets, they performed relatively poorly on the Eastern datasets, especially in recognition of negative emotions. While these results caution the use of these automatic methods of emotion recognition on non-Western faces, the results also suggest that the measurements of happiness outputted by these automatic methods are accurate and invariant across races and hence can still be utilized for cross-cultural studies of positive psychology.

Tracking and Detecting Depression Level using Facial Recognition and PEN&IQ Test

These Human facial expressions convey a lot of information visually rather than articulately. Facial expression recognition plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications including, but not limited to, human behaviour understanding, detection of mental disorders, and synthetic human expressions. Recognition of facial expression by computer with high recognition rate is still a challenging task. Two popular methods utilized mostly in the literature for the automatic FER systems are based on geometry and appearance. Facial Expression Recognition usually performed in four-stages consisting of pre-processing, face detection, feature extraction, and expression classification. The human face is an important part of an individual’s body and plays an important role in knowing the individual’s mood. The face is where a human expresses all his basic emotions. In the existing system, they examine the mental state manually by assessing them but have many disadvantages like we cannot predict any accurate solutions based on the assessment score because we might be not sure what kind of emotions the human user would be all time. To overcome this problem, a novel system is proposed to suggest an effective solution for predicting the mental state dynamically, we propose a hybrid architecture invoking facial based emotion sequence, PEN test and IQ test. By consistent monitoring of a human’s emotion and subjecting to PEN and IQ tests, the human’s mental state is routed. Combination of the above three techniques provides promising results for mental state and self-control

Development of Real-Time Emotion Recognition System Using Facial Expressions

This research presents a real-time emotion recognition system that combines human-friendly machine interaction with picture processing. For many years, facial detection has been available. Moving further, it is possible to simulate the emotions that people express on their faces and experience in their brains through the use of video, electric signals, or image forms. Since it is hard for computers to detect emotions from images or videos and a difficult task for the human eye, machine emotion detection requires a variety of image processing approaches for feature extraction. The approach proposed in this paper consists of two primary processes: facial expression recognition (FER) and face detection. The experimental investigation of facial emotion recognition is the main topic of this study. An emotion detection system's workflow consists of face detection, feature extraction, pre-processing, classification, and image acquisition. The emotion identification system uses the Haar cascade algorithm, an object detection algorithm, to recognize faces in an image or a real-time video, and the KNN Classifier for image classification in order to identify such emotions. Using the webcam to capture real-time photos, this system operates. The goal of this research is to develop an automatic facial expression recognition system that can recognize various emotions. Based on these studies, the system may be able to distinguish between a number of people who are fearful, furious, shocked, sad, or pleased, among other emotions.

A Bibliography Review on Facial Expression Recognition System by Conventional Approaches

The Facial Expression Recognition system is the process of identifying the emotional state of a person. It plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications but is not limited to, human behavior understanding, detection of mental disorders, and synthetic human expressions. Research in this field has drawn much attention given the continually increasing number of publications in the last 10 years. In this paper, bibliographic research was used and a comprehensive review of relevant publications was taken from the Scopus database. The research is defined as any research requiring information to be gathered from published materials. This bibliographic research gives descriptive information about a piece of work, such as author, title, date of publication, etc. The analysis also allows mapping of the scope and structure of discipline; locating the established collaboration patterns among countries. This analysis also identifies the current research interests and potential directions for future research.

Employing face reading technology to study the effects of the fake news' message sidedness on consumers' emotional valence and arousal

PurposeThis study aims to investigate how message sidedness affects the impact of fake news posted on social media on consumers' emotional responses.Design/methodology/approachThe study involves a face-tracking experiment in which 198 participants were exposed to different fake news messages concerning the COVID-19 vaccine. Specifically, participants were exposed to fake news using (1) a one-sided negative fake news message in which the message was entirely unfavorable and (2) a two-sided fake news message in which the negative message was mixed with favorable information. Noldus FaceReader 7, an automatic facial expression recognition system, was used to recognize participants' emotions as they read fake news. The authors sampled 17,450 observations of participants' emotional responses.FindingsThe results provide evidence of the significant influence of message sidedness on consumers' emotional valence and arousal. Specifically, two-sided fake news positively influences emotional valence, while one-sided fake news positively influences emotional arousal.Originality/valueThe current study demonstrates that research on fake news posted on social media may particularly benefit from insights regarding the potential but often overlooked importance of strategic design choices in fake news messages and their impact on consumers' emotional responses.

Parameter optimization of histogram-based local descriptors for facial expression recognition.

An important task in automatic facial expression recognition (FER) is to describe facial image features effectively and efficiently. Facial expression descriptors must be robust to variable scales, illumination changes, face view, and noise. This article studies the application of spatially modified local descriptors to extract robust features for facial expressions recognition. The experiments are carried out in two phases: firstly, we motivate the need for face registration by comparing the extraction of features from registered and non-registered faces, and secondly, four local descriptors (Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), Compound Local Binary Patterns (CLBP), and Weber's Local Descriptor (WLD)) are optimized by finding the best parameter values for their extraction. Our study reveals that face registration is an important step that can improve the recognition rate of FER systems. We also highlight that a suitable parameter selection can increase the performance of existing local descriptors as compared with state-of-the-art approaches.

Hybrid Features Extraction for Emotion Detection using Deep Convolutional Neural Network

Abstract: These Human facial expressions communicate much information more visually than verbally. Recognition of facial expressions is essential for human-machine interaction. Applications for automatic facial expression recognition include but are not limited to, comprehending human behaviour, spotting mental illnesses, and creating artificial human emotions. Humancomputer interaction, autonomous cars, and a wide range of multimedia applications all rely heavily on facial expression detection. In this study, we present a reusable architecture for recognizing human emotions based on facial expressions. Two machine learning algorithms that may be learned in advance for use in real-time applications make up the framework. At first, we use AdaBoost cascade classifiers to find instances of faces in the photos. The extracted neighborhood difference characteristics then serve as a representation of a face's features according to its localized appearance data. Instead of only looking at intensity data, the NDF predicts multiple patterns depending on the connections between surrounding areas. Despite a high identification rate, facial expression recognition by computers remains difficult. Based on geometry and appearance, two widely used approaches for automated FER systems are often used in the literature. Pre-processing, face detection, feature extraction, and expression classification are the four steps that typically make up facial expression recognition. Various deep learning techniques (convolutional neural networks) were used in this effort to recognize the primary seven human emotions: anger, fear, happiness, surprise, and neutrality

Facial Expression Recognition with Contrastive Learning and Uncertainty-Guided Relabeling

Facial expression recognition (FER) plays a vital role in the field of human-computer interaction. To achieve automatic FER, various approaches based on deep learning (DL) have been presented. However, most of them lack for the extraction of discriminative expression semantic information and suffer from the problem of annotation ambiguity. In this paper, we propose an elaborately designed end-to-end recognition network with contrastive learning and uncertainty-guided relabeling, to recognize facial expressions efficiently and accurately, as well as to alleviate the impact of annotation ambiguity. Specifically, a supervised contrastive loss (SCL) is introduced to promote inter-class separability and intra-class compactness, thus helping the network extract fine-grained discriminative expression features. As for the annotation ambiguity problem, we present an uncertainty estimation-based relabeling module (UERM) to estimate the uncertainty of each sample and relabel the unreliable ones. In addition, to deal with the padding erosion problem, we embed an amending representation module (ARM) into the recognition network. Experimental results on three public benchmarks demonstrate that our proposed method facilitates the recognition performance remarkably with 90.91% on RAF-DB, 88.59% on FERPlus and 61.00% on AffectNet, outperforming current state-of-the-art (SOTA) FER methods. Code will be available at http//github.com/xiaohu-run/fer_supCon .

Intelligent Multi-Criteria Decision Support for Renovation Solutions for a Building Based on Emotion Recognition by Applying the COPRAS Method and BIM Integration

With accelerating climate change and the urgent need to cut carbon emissions, global focus has turned to the existing building stock and its renovation. Sustainable renovation helps to achieve better energy performance and gain wider sustainability benefits, such as increased value of a building, improved indoor and outdoor comfort, reduced carbon emissions, and the higher satisfaction and better emotional state of inhabitants. Numerous systems and tools have been developed worldwide to assist with decision making in the choice of preferred modernisation scenarios and alternatives. However, social aspects are often neglected in the existing systems, and emotions of inhabitants are rarely analysed. To close this gap, the present study proposes an innovative decision-making framework for sustainable renovation solutions, based on emotion recognition. The framework makes it possible to assess various renovation alternatives against sustainability criteria and real-time measurements of the emotional states of inhabitants. Based on the proposed framework, an intelligent multi-criteria decision support system was developed by integrating COPRAS and the facial action coding system, the method of automatic facial expression recognition, and the continuous calibration and participant methods. The system was tested in the case study of renovation solutions for a building located in Ukraine. The research results revealed that the proposed renovation solutions had a positive impact on the emotional state of inhabitants, especially when visual materials such as drawings were presented. Some case studies were analysed together with the application of decision system tools and building information modelling (BIM) subsystem integration as a multidiscipline application of various applied sciences for representation and data analysis. The authors of this research have been analysing human emotional, affective and physiological states for many years and collected over a billion of these data in Vilnius city during the H2020 ROCK, SAVAS and BIM4REN projects. Data acquired during measurements in Vilnius were used to determine correlations and trends for the case study. The proposed methodology and findings of the study can be useful for researchers who use the evaluation and analysis of human emotions when there is a need to choose appropriate renovation measures or find alternative solutions.

An automatic improved facial expression recognition for masked faces

Automatic facial expression recognition (AFER), sometimes referred to as emotional recognition, is important for socializing. Automatic methods in the past two years faced challenges due to Covid-19 and the vital wearing of a mask. Machine learning techniques tremendously increase the amount of data processed and achieved good results in such AFER to detect emotions; however, those techniques are not designed for masked faces and thus achieved poor recognition. This paper introduces a hybrid convolutional neural network aided by a local binary pattern to extract features in an accurate way, especially for masked faces. The basic seven emotions classified into anger, happiness, sadness, surprise, contempt, disgust, and fear are to be recognized. The proposed method is applied on two datasets: the first represents CK and CK +, while the second represents M-LFW-FER. Obtained results show that emotion recognition with a face mask achieved an accuracy of 70.76% on three emotions. Results are compared to existing techniques and show significant improvement.

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.

Multi-Class Facial Emotion Recognition Using Hybrid Dense Squeeze Network

Automatic facial expression recognition (FER) is utilized in various applications like psychoanalysis, intelligent driving, robot manufacturing, etc. Numerous researchers have been looking for better techniques to improve the accuracy of FER. In fact, FER under laboratory conditions has almost achieved top accuracy. Besides, label deviations or errors caused by annotators’ subjectivity also make the FER task much tougher. Thus, more and more researchers begin to find new ways to handle with the FER problems. In this work, a new deep learning (DL) model called dense squeeze network with improved red deer optimization (DenseSNet_IRDO) is proposed for the recognition of facial emotions. The steps used for FER are pre-processing, fused deep feature extraction-selection and classification. Initially, the facial images are pre-processed using improved trilateral filter (ITF) for improving the quality of images. Next, the fusion of feature extraction and selection is performed using the DenseSNet. Here the extraction of deep features is done with the dense network and the relevant features are selected with the squeeze network. Finally, the last layer of squeeze network performs the classification of various facial emotions. Here, the loss in the classification is optimized using IRDO. This DenseSNet_IRDO architecture is more robust and avoids overfitting that occurs while training the small dataset. The datasets used in this work are CK[Formula: see text], JAFEE and FERFIN. The proposed FER classification using datasets CK[Formula: see text], JAFEE and FERFIN with DenseSNet_IRDO model achieved the accuracy of 99.91%, 99.90% and 99.89%, respectively. Thus, the proposed DenseSNet_IRDO classifier model obtained higher accuracy in the detection of FER than other methods.

Improved Methods for Automatic Facial Expression Recognition

Facial expressions constitute one of the most effective and instinctive methods that allow people to communicate their emotions and intentions. In this context, the both Machine Learning (ML) and Convolutional Neural Networks (CNNs) have been used for emotion recognition. Efficient recognition systems are required for good human-computer interaction. However, facial expression recognition is related to several methods that impact the performance of facial recognition systems. In this paper, we demonstrate a state-of-the-art of 65% accuracy on the FER2013 dataset by leveraging numerous techniques from recent research and we also proposed some new methods for improving accuracy by combining CNN architectures such as VGG-16 and Resnet-50 with auxiliary datasets such as JAFFE and CK. To predict emotions, we used a second approach based on geometric features and facial landmarks to calculate and transmit the feature vector to the SVM model. The results show that the ResNet50 model outperforms all other emotion prediction models in real time by maximizing.

A feature boosted deep learning method for automatic facial expression recognition.

Automatic facial expression recognition (FER) plays a crucial role in human-computer based applications such as psychiatric treatment, classroom assessment, surveillance systems, and many others. However, automatic FER is challenging in real-time environment. The traditional methods used handcrafted methods for FER but mostly failed to produce superior results in the wild environment. In this regard, a deep learning-based FER approach with minimal parameters is proposed, which gives better results for lab-controlled and wild datasets. The method uses features boosting module with skip connections which help to focus on expression-specific features. The proposed approach is applied to FER-2013 (wild dataset), JAFFE (lab-controlled), and CK+ (lab-controlled) datasets which achieve accuracy of 70.21%, 96.16%, and 96.52%. The observed experimental results demonstrate that the proposed method outperforms the other related research concerning accuracy and time.

Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling

Facial expressions reflect people’s feelings, emotions, and motives, attracting researchers to develop a self-acting automatic facial expression recognition system. With the advances of deep learning frameworks for automatic facial expression recognition, the model complexity, limited training samples, and subtle micro facial muscle movements make the facial emotion expression system challenging. This research proposed a deep learning framework using fine-grained facial action unit detection to identify facial activity, behavior, and mood and recognize a person’s emotions based on these individual patterns. The proposed facial expression recognition system involves pre-processing, feature representation and normalization, hyper-parameter tuning, and classification. Here, two different convolutional neural network models have been introduced because of feature learning and representation, followed by classification. Various advanced feature representation methods, such as image augmentation, matrix normalization, fine-tuning, and transfer learning methods, have been applied to improve the performance of the proposed work. The proposed work’s performance and efficiency are evaluated under different approaches. The proposed work has been tested on standard Static Facial Expressions in the Wild, short name SFEW 1.0, SFEW 2.0, and Indian Movie Face (IMFDB) benchmark databases. The performances of the proposed system due to these databases are 48.15%, 80.34%, and 64.17%, respectively. The quantitative analysis of these results is compared with the standard existing state-of-the-art methods that show the proposed model outperforms the other competing methods.