Facial Expression Recognition Algorithm Research Articles

Human Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM

Human emotion recognition is crucial in various technological domains, reflecting our growing reliance on technology. Facial expressions play a vital role in conveying and preserving human emotions. While deep learning has been successful in recognizing emotions in video sequences, it struggles to effectively model spatio-temporal interactions and identify salient features, limiting its accuracy. This research paper proposed an innovative algorithm for facial expression recognition which combined a deep learning algorithm and dynamic texture methods. In the initial phase of this study, facial features were extracted using the Visual-Geometry-Group (VGG19) model and input into Long-Short-Term-Memory (LSTM) cells to capture spatio-temporal information. Additionally, the HOG-HOF descriptor was utilized to extract dynamic features from video sequences, capturing changes in facial appearance over time. Combining these models using the Multimodal-Compact-Bilinear (MCB) model resulted in an effective descriptor vector. This vector was then classified using a Support Vector Machine (SVM) classifier, chosen for its simpler interpretability compared to deep learning models. This choice facilitates better understanding of the decision-making process behind emotion classification. In the experimental phase, the fusion method outperformed existing state-of-the-art methods on the eNTERFACE05 database, with an improvement margin of approximately 1%. In summary, the proposed approach exhibited superior accuracy and robust detection capabilities.

Recognizing facial emotion leveraging convolutional neural network

Facial expression recognition technology is a technology based on the combination of artificial intelligence technology and biology, and it is interdisciplinary research. The appearance of this technology shows the diversity of the development direction and application fields of computer technology, but it also means that the development of facial expression recognition technology needs not only the support of computer technology but also the exploration and progress of biology. At present, to recognize facial expressions a system usually contains three stages, namely, face detection stage, feature extraction stage, and face expression recognition stage. The facial expression recognition algorithm is transformed into a classification problem, and a convolutional neural network (CNN) is adopted, where features of the facial expressions could be automatically extracted. It takes as input a preprocessed image and can directly produce encoded features and predictions, leveraging the end-to-end strategy. It could discard the complicated intermediate modeling process of traditional machine learning. In this work a CNN is implemented for the recognition of facial expression on Fer2013 dataset. Moreover, the effectiveness of different numbers of CNN layers are testified on this problem. It could be concluded that with deeper architectures, the CNN tends to perform better.

Development of a Universal Validation Protocol and an Open-Source Database for Multi-Contextual Facial Expression Recognition.

Facial expression recognition (FER) poses a complex challenge due to diverse factors such as facial morphology variations, lighting conditions, and cultural nuances in emotion representation. To address these hurdles, specific FER algorithms leverage advanced data analysis for inferring emotional states from facial expressions. In this study, we introduce a universal validation methodology assessing any FER algorithm's performance through a web application where subjects respond to emotive images. We present the labelled data database, FeelPix, generated from facial landmark coordinates during FER algorithm validation. FeelPix is available to train and test generic FER algorithms, accurately identifying users' facial expressions. A testing algorithm classifies emotions based on FeelPix data, ensuring its reliability. Designed as a computationally lightweight solution, it finds applications in online systems. Our contribution improves facial expression recognition, enabling the identification and interpretation of emotions associated with facial expressions, offering profound insights into individuals' emotional reactions. This contribution has implications for healthcare, security, human-computer interaction, and entertainment.

A dynamic fusion of features from deep learning and the HOG-TOP algorithm for facial expression recognition

A dynamic fusion of features from deep learning and the HOG-TOP algorithm for facial expression recognition

Analysis and Recognition of Voluntary Facial Expression Mimicry Based on Depressed Patients.

Many clinical studies have shown that facial expression recognition and cognitive function are impaired in depressed patients. Different from spontaneous facial expression mimicry (SFEM), 164 subjects (82 in a case group and 82 in a control group) participated in our voluntary facial expression mimicry (VFEM) experiment using expressions of neutrality, anger, disgust, fear, happiness, sadness and surprise. Our research is as follows. First, we collected a large amount of subject data for VFEM. Second, we extracted the geometric features of subject facial expression images for VFEM and used Spearman correlation analysis, a random forest, and logistic regression-based recursive feature elimination (LR-RFE) to perform feature selection. The features selected revealed the difference between the case group and the control group. Third, we combined geometric features with the original images and improved the advanced deep learning facial expression recognition (FER) algorithms in different systems. We propose the E-ViT and E-ResNet based on VFEM. The accuracies and F1 scores were higher than those of the baseline models, respectively. Our research proved that it is effective to use feature selection to screen geometric features and combine them with a deep learning model for depression facial expression recognition.

A high‐performance and lightweight framework for real‐time facial expression recognition

AbstractFacial expression recognition technology has become a powerful tool for conveying human emotions and intentions and is widely used in areas such as assisted driving and intelligent medical care. Due to the limited computing power of current hardware devices and the real‐time requirements of application scenarios, this paper proposes a high‐performance and lightweight framework for real‐time facial expression recognition framework to solve the problem of real‐time completion of expression recognition tasks under low hardware costs. To address these issues, this paper first designs a RepVGG and mobileNetV2 dual‐channel structure in the feature extraction. It is then input into the MobileViT Block for global feature modelling. Finally, the position vector of the capsule network is used to replace the output of the global pooling, preserving the spatial relationship of the salient features and enhancing the classification effect. Compared with the mainstream facial expression recognition algorithm that cannot get good classification results under low complexity conditions, the model has a significant accuracy improvement while ensuring lightweight. With only 294.60M FLOPS and 0.95M parameters, it achieved an accuracy of 97.53% on the KDEF dataset and 85.56% on the RAF‐DB, demonstrating the advanced nature of the algorithm.

A facial expression recognition algorithm incorporating SVM and explainable residual neural network

A facial expression recognition algorithm incorporating SVM and explainable residual neural network

Deep Learning Approach for Facial Expression Recognition

Abstract: In recent decades, facial expression recognition has emerged as a hot topic with significant implications in the realm of human-computer interaction. The simplest way for humans to express their emotions is through facial expressions. Non-verbal communication relies heavily on facial expression. This study outlines deep learning-based Facial Expression Recognition (FER) algorithm. The performance of the FER approach is compared based on the number of expressions detected and the difficulty of CNN algorithms. The FER2013, CK+ databases were used in testing the design. CNNs (Convolutional Neural Networks) have as of late gained fame in the field of profound learning because of its superb plan and capacity to convey clever outcomes without the requirement for manual feature extraction from raw information. The suggested algorithm achieves a higher rate of recognition on four datasets.

Adaptive graph regularized transferable regression for facial expression recognition

Facial expression recognition (FER) has tremendous potential in affective computing and human-computer interaction fields. Traditional FER algorithms usually perform the training and testing of models on a single domain. However, face images in real environments usually come from different domains, which greatly limits the performance of traditional algorithms. To handle this cross-domain recognition problem, we put forward a novel transfer learning model, called adaptive graph regularized transferable regression (AGTR), which can learn a discriminative projection matrix by embedding relaxed label regression, class sparsity structure, and adaptive graph structure into a unified framework. To be specific, in our method, we develop a relaxed label regression to learn a projection matrix. Then, we exploit a class sparsity structure in each class of samples separately to obtain a consistent subspace. Further, we design a novel adaptive graph structure, which can adaptively discover the geometric relationship between samples. Finally, we verify the advancement of our approach on four public facial expression databases.

A novel facial expression recognition algorithm using geometry β –skeleton in fusion based on deep CNN

Facial expression recognition (FER) methods based on single-source facial data often suffer from reduced accuracy or unpredictability due to facial occlusion or illumination changes. To address this, a new technique called Fusion-CNN is proposed. It improves accuracy by extracting hybrid features using a β-skeleton undirected graph and an ellipse with parameters trained using a 1D-CNN. In addition, a 2D-CNN is trained on the same image. The outputs from these two subnetworks are fused, and their features are concatenated to create a feature vector for classification in a deep neural network. The proposed method is evaluated on four public face datasets: the extended Cohn-Kanade (CK+) dataset, the Japanese Female Facial Expression (JAFFE) dataset, Karolinska Directed Emotional Faces (KDEF), and Oulu-CASIA. The experimental results show that Fusion-CNN outperforms other algorithms, achieving recognition accuracy of 98.22%, 93.07%, 90.30%, and 90.13% for the CK+, JAFFE, KDEF, and Oulu-CASIA datasets, respectively.

A CNN-Based Facial Expression Recognition System

Facial expression recognition has long attracted researchers from a variety of domains, such as human-computer interaction, emotion analysis, intelligent medical care, and so on. Various human-designed features have been used in FER to extract image appearance features, but human-designed methods are difficult to be used to recognize new face images, which poses challenges for FER in uncontrolled environments. Recently, CNNs and FNNs have been used for facial expression recognition. The new CNN-based recognition approach yields excellent FER results. The numerous convolution and pooling layers that CNN possesses allow it to extract higher and multi-level features of the entire face or local regions. Additionally, CNN has excellent classification performance when it comes to the feature extraction of facial expression pictures. However, the mechanism through which FNN remains largely controversial. In this work, the experiments are carried out on the comparison of three facial expression recognition algorithms: K-Nearest Neighbor (KNN), Support Vector Machine (SVM), and Convolutional Neural Network (CNN), and the experimental results show that, using CNN can get more stable recommendation results of better quality.

Learning Transferable Sparse Representations for Cross-Corpus Facial Expression Recognition

An assumption widely used in traditional facial expression recognition algorithms is that the training and testing are conducted on the same dataset. However, this assumption does not hold in practice, in which the training data and testing data are often from different datasets. In this scenario, directly deploying these algorithms would lead to severe information loss and performance degradation due to the domain shift. To address this challenging problem, in this paper, we propose a novel transferable sparse subspace representation method (TSSR) for cross-corpus facial expression recognition. Specifically, in order to reduce the cross-corpus mismatch, inspired by sparse subspace clustering, we advocate reconstructing the source and target samples using the source data points based on L1-norm sparse representation. Each data point in source and target corpora can be ideally represented as a combination of a few other source points from its own subspace. Moreover, we take into account the local geometrical information within the cross-corpus data by adopting a graph Laplacian regularizer, which can efficiently preserve the local manifold structure and better transfer knowledge between two corpora. Finally, experimental results on several facial expression datasets demonstrate the superiority of the proposed method over some state-of-the-art methods.

Face expression recognition based on NGO-BILSTM model.

Facial expression recognition has always been a hot topic in computer vision and artificial intelligence. In recent years, deep learning models have achieved good results in accurately recognizing facial expressions. BILSTM network is such a model. However, the BILSTM network's performance depends largely on its hyperparameters, which is a challenge for optimization. In this paper, a Northern Goshawk optimization (NGO) algorithm is proposed to optimize the hyperparameters of BILSTM network for facial expression recognition. The proposed methods were evaluated and compared with other methods on the FER2013, FERplus and RAF-DB datasets, taking into account factors such as cultural background, race and gender. The results show that the recognition accuracy of the model on FER2013 and FERPlus data sets is much higher than that of the traditional VGG16 network. The recognition accuracy is 89.72% on the RAF-DB dataset, which is 5.45, 9.63, 7.36, and 3.18% higher than that of the proposed facial expression recognition algorithms DLP-CNN, gACNN, pACNN, and LDL-ALSG in recent 2 years, respectively. In conclusion, NGO algorithm effectively optimized the hyperparameters of BILSTM network, improved the performance of facial expression recognition, and provided a new method for the hyperparameter optimization of BILSTM network for facial expression recognition.

Performance Evaluation of Fusion Based Efficient Algorithm for Facial Expression Recognition

Objectives: To develop face expression recognition system using JAFFE database and to evaluate the performance of the face expression recognition models. Methods: This study used the FER model based on modified-HoG (Histogram of oriented gradient), LBP (Local Binary Patterns) and Fast Key point detector and BRIEF descriptor (FKBD) to extract the significant features of JAFFE dataset. The features extracted using HoG, LBP and FKBD techniques form a feature vector. Then, the fusion of all the features is carried out at the feature level. The multiclass SVM and KNN classifiers are used to recognize the facial expressions, effectively. Findings: In this work, an effort is made to develop a robust FER model using JAFFE database. It is recorded that, based on the experimental results, the proposed model suits better with a performance rate of 98.26% for SVM and 96.51% for KNN, when compared with the different state-of-the-art methods. Novelty: Many FER models have been developed and adopted for enhancing their quality and to extract the facial features using transform and frequency domains. It is observed that, maximum approaches are based on generating the texture features. The fusion at the feature level using modified HoG, LBP and FKBD is performed and the SVM model is more compatible when compared with other classifiers and it supports one-to-one and one-to-many comparisons’ technique. Keywords: Face Expression Recognition; Local Binary Pattern; Emotions; Nearest Neighbor; Histogram of Gradients

Statement of Removal

ABSTRACT Aiming at the problem that uneven illumination easily affects the facial expression recognition rate and reduces the facial expression recognition rate, a facial expression recognition algorithm based on local features and deep belief network (DBN) is proposed. Firstly, the non-uniform illumination invariant feature of LSH in facial expression image is extracted; Secondly, Gauss Laplace operator (log) is used to extract the edge detail features of facial expression; Then, the facial expression features fused by LSH and log are constructed; Finally, the DBN model is used to train the facial expression features fused by LSH and log, and the trained DBN model is used for facial expression recognition. Experiments on Jaffe facial expression database and Uighur facial expression database show that the algorithm has strong robustness and effectiveness.

Energetic optimization of an autonomous mobile socially assistive robot for autism spectrum disorder.

The usage of socially assistive robots for autism therapies has increased in recent years. This novel therapeutic tool allows the specialist to keep track of the improvement in socially assistive tasks for autistic children, who hypothetically prefer object-based over human interactions. These kinds of tools also allow the collection of new information to early diagnose neurodevelopment disabilities. This work presents the integration of an output feedback adaptive controller for trajectory tracking and energetic autonomy of a mobile socially assistive robot for autism spectrum disorder under an event-driven control scheme. The proposed implementation integrates facial expression and emotion recognition algorithms to detect the emotions and identities of users (providing robustness to the algorithm since it automatically generates the missing input parameters, which allows it to complete the recognition) to detonate a set of adequate trajectories. The algorithmic implementation for the proposed socially assistive robot is presented and implemented in the Linux-based Robot Operating System. It is considered that the optimization of energetic consumption of the proposal is the main contribution of this work, as it will allow therapists to extend and adapt sessions with autistic children. The experiment that validates the energetic optimization of the proposed integration of an event-driven control scheme is presented.

Facial expression recognition based on improved residual network

AbstractFacial expressions are an important part of human emotional signals and their recognition has become an important topic of research in the field of pattern recognition. Deep learning based methods have achieved great success in the recognition of facial expressions. However, with the evolution of convolution neural networks and the increased network depth, these methods suffer from problems such as degraded network performance and loss of feature information. To address these problems, a novel facial expression recognition algorithm based on an improved residual neural network is proposed. First, a residual neural network is designed to extract deep features while retaining the shallow ones. This can effectively prevent the degradation of network performance. Moreover, when the gradient of the Rectified Linear Units activation function used in the residual module is 0, it will inactivate the neurons and cause a loss of feature information. To address this, the Mish activation function is used instead. The slight allowance for negative values in Mish improves the gradient flow. Next, an inception module is introduced to obtain richer feature information under the same receptive field. Finally, by conducting verification experiments on the public datasets CK+ and KDEF, the authors manage to solve the problems of degraded network performance and insufficient information from extracted features, achieving recognition accuracy rates of 96.37% and 93.38% on the two datasets, respectively.

Facial Expression Recognition Based on Multi-Channel Attention Residual燦etwork

For the problems of complex model structure and too many training parameters in facial expression recognition algorithms, we proposed a residual network structure with a multi-headed channel attention (MCA) module. The migration learning algorithm is used to pre-train the convolutional layer parameters and mitigate the overfitting caused by the insufficient number of training samples. The designed MCA module is integrated into the ResNet18 backbone network. The attention mechanism highlights important information and suppresses irrelevant information by assigning different coefficients or weights, and the multi-head structure focuses more on the local features of the pictures, which improves the efficiency of facial expression recognition. Experimental results demonstrate that the model proposed in this paper achieves excellent recognition results in Fer2013, CK+ and Jaffe datasets, with accuracy rates of 72.7%, 98.8% and 93.33%, respectively.

Comparing the Robustness of Humans and Deep Neural Networks on Facial Expression Recognition

Emotion recognition, and more particularly facial expression recognition (FER), has been extensively used for various applications (e.g., human–computer interactions). The ability to automatically recognize facial expressions has been facilitated with recent progress in the fields of computer vision and artificial intelligence. Nonetheless, FER algorithms still seem to face difficulties with image degradations due to real-life conditions (e.g., because of image compression or transmission). In this paper, we propose to investigate the impact of different distortion configurations on a large number of images of faces on human performance, thanks to the conduct of a crowdsourcing experiment. We further compare human performance with two open-source FER algorithms. Results show that, overall, models are more sensitive to distortions than humans—even when fine-tuned. Furthermore, we broach the subject of annotation errors and bias which exist in several well-established datasets, and suggest approaches to improve the latter.

Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning.

Facial expression recognition (FER) has received significant attention in the past decade with witnessed progress, but data inconsistencies among different FER datasets greatly hinder the generalization ability of the models learned on one dataset to another. Recently, a series of cross-domain FER algorithms (CD-FERs) have been extensively developed to address this issue. Although each declares to achieve superior performance, comprehensive and fair comparisons are lacking due to inconsistent choices of the source/target datasets and feature extractors. In this work, we first propose to construct a unified CD-FER evaluation benchmark, in which we re-implement the well-performing CD-FER and recently published general domain adaptation algorithms and ensure that all these algorithms adopt the same source/target datasets and feature extractors for fair CD-FER evaluations. Based on the analysis, we find that most of the current state-of-the-art algorithms use adversarial learning mechanisms that aim to learn holistic domain-invariant features to mitigate domain shifts. However, these algorithms ignore local features, which are more transferable across different datasets and carry more detailed content for fine-grained adaptation. Therefore, we develop a novel adversarial graph representation adaptation (AGRA) framework that integrates graph representation propagation with adversarial learning to realize effective cross-domain holistic-local feature co-adaptation. Specifically, our framework first builds two graphs to correlate holistic and local regions within each domain and across different domains, respectively. Then, it extracts holistic-local features from the input image and uses learnable per-class statistical distributions to initialize the corresponding graph nodes. Finally, two stacked graph convolution networks (GCNs) are adopted to propagate holistic-local features within each domain to explore their interaction and across different domains for holistic-local feature co-adaptation. In this way, the AGRA framework can adaptively learn fine-grained domain-invariant features and thus facilitate cross-domain expression recognition. We conduct extensive and fair comparisons on the unified evaluation benchmark and show that the proposed AGRA framework outperforms previous state-of-the-art methods.