Facial Expression Recognition Problem Research Articles

Common Latent Embedding Space for Cross-Domain Facial Expression Recognition

In practical facial expression recognition (FER), the training data and test data are often obtained from different domains. It is obvious that the domain disparity could significantly degrade the recognition performance. To tackle this challenging cross-domain FER problem, we put forward a novel method termed common latent embedding space (CLES). To be specific, first, we obtain a common embedding space for cross-domain samples by matrix factorization (MF). Then, the dual-graph Laplacian is applied to this common embedding space to narrow the gap across distinct domains and, meanwhile, explores the inherent geometric information. Furthermore, to characterize the global relationship of the cross-domain samples, the self-representation strategy is used to guide the learning of the common embedding space. Finally, comprehensive experiments on four benchmark databases indicate that the proposed method can achieve better performance in comparison with the state-of-the-art methods on cross-domain FER tasks.

Learning facial expression and body gesture visual information for video emotion recognition

Recent research has shown that facial expressions and body gestures are two significant implications in identifying human emotions. However, these studies mainly focus on contextual information of adjacent frames, and rarely explore the spatio-temporal relationships between distant or global frames. In this paper, we revisit the facial expression and body gesture emotion recognition problems, and propose to improve the performance of video emotion recognition by extracting the spatio-temporal features via further encoding temporal information. Specifically, for facial expression, we propose a super image-based spatio-temporal convolutional model (SISTCM) and a two-stream LSTM model to capture the local spatio-temporal features and learn global temporal cues of emotion changes. For body gestures, a novel representation method and an attention-based channel-wise convolutional model (ACCM) are introduced to learn key joints features and independent characteristics of each joint. Extensive experiments on five common datasets are carried out to prove the superiority of the proposed method, and the results proved learning two visual information leads to significant improvement over the existing state-of-the-art methods.

FgbCNN: A unified bilinear architecture for learning a fine-grained feature representation in facial expression recognition

A facial expression recognition system has been proposed in this paper. The challenges of the facial expression recognition system lie due to low intra-class variance within a class of negative emotions, such as anger, disgust, and fear. A conventional Convolution Neural Network (CNN) model may extract discriminative features and has shown outstanding performance in various computer vision tasks. However, it is unable to extract the second-order feature information that demonstrates the interaction between features in the image of wild-environment datasets. This paper presents a novel method to solve the facial expression recognition problem by addressing several limitations concerning emotion recognition problems. A deep learning-based bilinear convolutional neural network framework has been proposed, termed an Fine-Grained Bilinear CNN (FgbCNN) model that consists of two branches with optimized CNN along with a normalization layer composed of batch-normalization, square-root normalization, L2-normalization, and drop-out layers. Here, local and holistic features have been aggregated using a dot-product layer to extract more discriminant features. Finally, experimenting with two wild-environments (SFEW 1.0 and SFEW 2.0) and two lab-controlled datasets (KDEF and CK+), it has been observed that the proposed model can minimize the intra-class variances and has attained outstanding performance compared to other state-of-the-art methods.

Multi-Class Facial Emotion Recognition Using Hybrid Dense Squeeze Network

Automatic facial expression recognition (FER) is utilized in various applications like psychoanalysis, intelligent driving, robot manufacturing, etc. Numerous researchers have been looking for better techniques to improve the accuracy of FER. In fact, FER under laboratory conditions has almost achieved top accuracy. Besides, label deviations or errors caused by annotators’ subjectivity also make the FER task much tougher. Thus, more and more researchers begin to find new ways to handle with the FER problems. In this work, a new deep learning (DL) model called dense squeeze network with improved red deer optimization (DenseSNet_IRDO) is proposed for the recognition of facial emotions. The steps used for FER are pre-processing, fused deep feature extraction-selection and classification. Initially, the facial images are pre-processed using improved trilateral filter (ITF) for improving the quality of images. Next, the fusion of feature extraction and selection is performed using the DenseSNet. Here the extraction of deep features is done with the dense network and the relevant features are selected with the squeeze network. Finally, the last layer of squeeze network performs the classification of various facial emotions. Here, the loss in the classification is optimized using IRDO. This DenseSNet_IRDO architecture is more robust and avoids overfitting that occurs while training the small dataset. The datasets used in this work are CK[Formula: see text], JAFEE and FERFIN. The proposed FER classification using datasets CK[Formula: see text], JAFEE and FERFIN with DenseSNet_IRDO model achieved the accuracy of 99.91%, 99.90% and 99.89%, respectively. Thus, the proposed DenseSNet_IRDO classifier model obtained higher accuracy in the detection of FER than other methods.

Comparison of Different Depth of Convolutional Neural Network Deep and shallow CNN comparison based on FER-2013

The purpose of this paper is to cross-compare the different nature of the decision process of convolutional neural networks (CNNs) of different depths, as well as the characteristics of the ensemble model. The Facial Expression Recognition (FER) problem is selected as the carrier of this experiment. The dataset chosen is FER-2013, one of the most challenging FER datasets due to its complicated and natural contents. This research first trained four models that have different architectures using FER-2013. One is the shallow convolutional neural network, and the other three are deep pre-trained CNNs, including ResNet50 with weights from ImageNet, VGG16 with weights from ImageNet, and VGG16 with weights from VGGFaceNet. Then, by using a gradient-based Class activation mapping technique as a visualization technique, this experiment successfully displayed the different decision-making processes of the selected models and illustrated how the depth of the neural network influences the feature extraction process. This work further experimented with convolutional neural networks(CNNs) having different depths by varying ensemble combinations. Finally, all three-model combinations were tested, and the experiment results show that models that are more different in architecture would result in a better ensemble performance than those with a similar architecture. Thus, one of the inspirations we could get from this work is that models with dramatically different architectures could earn a more remarkable improvement for future ensemble models.

Performance Analysis and Evaluation of Different Deep Learning Algorithms for Facial Expression Recognition

Emotions are dynamic biological states that are connected to all of the nerve systems. The problem of facial expression recognition has been thoroughly investigated, leading to the development of some robust and accurate face recognition algorithms. The effectiveness of three such algorithms (CNN, VGG16, and ResNet50) that have been widely studied and applied in the research community are investigated and compared in this paper. The aim is to use grayscale images to train these training models and compare their accuracy and data losses. The system will be able to detect the seven facial expressions Angry, Neutral, Contempt, Disgust, Fear, Happy, and Sad after training these models. To compare their precision, the same batch size and epoch were used. After reviewing all possible evaluations based on these output matrices, it is clear that all three networks produce reliable effect identification, with CNN being the most accurate.

A Comparative Study of Local Descriptors and Classifiers for Facial Expression Recognition

Facial Expression Recognition (FER) is a growing area of research due to its numerous applications in market research, video gaming, healthcare, security, e-learning, and robotics. One of the most common frameworks for recognizing facial expressions is by extracting facial features from an image and classifying them as one of several prototypic expressions. Despite the recent advances, it is still a challenging task to develop robust facial expression descriptors. This study aimed to analyze the performances of various local descriptors and classifiers in the FER problem. Several experiments were conducted under different settings, such as varied extraction parameters, different numbers of expressions, and two datasets, to discover the best combinations of local descriptors and classifiers. Of all the considered descriptors, HOG (Histogram of Oriented Gradients) and ALDP (Angled Local Directional Patterns) were some of the most promising, while SVM (Support Vector Machines) and MLP (Multi-Layer Perceptron) were the best among the considered classifiers. The results obtained signify that conventional FER approaches are still comparable to state-of-the-art methods based on deep learning.

Patch attention convolutional vision transformer for facial expression recognition with occlusion

Despite substantial progress in Facial Expression Recognition (FER) in recent decades, most previous methods have been developed to recognize constrained facial expressions. Real-world occlusions lead to invisible facial regions and contaminated facial features, which undoubtedly increase the difficulty of FER in the wild. Therefore, a Patch Attention Convolutional Vision Transformer (PACVT) is proposed to tackle the occlusion FER problem. The backbone convolutional neural network is used to extract facial feature maps, which are cropped into multiple regional patches to extract local and global features. The Patch Attention Unit (PAU) is designed to perceive occluded regions by adaptively calculating the patch-level attention weights of local features for expression recognition. The facial patches are mapped into sequences of visual tokens, and the Vision Transformer (ViT) is employed to capture the interactions and correlations between these visual tokens from a global perspective. The self-attention in ViT enables the PACVT to focus on the salient patches with discriminative features and ignore the occlusion. Experiments are conducted on three widely used expression datasets and their occlusion subsets, and the results demonstrate that the proposed PACVT outperforms state-of-the-art methods on occlusion FER. Cross-dataset experiment results evidence the generalization ability of the PACVT.

FLEPNet: Feature Level Ensemble Parallel Network for Facial Expression Recognition

With the advent of deep learning, the research on facial expression recognition (FER) has received a lot of interest. Different deep convolutional neural network (DCNN) architectures have been developed for real-time and efficient FER. One of the challenges in FER is obtaining trustworthy features that are strongly associated with facial expression changes. Furthermore, traditional DCNNs for FER problems have two significant issues: insufficient training data, which leads to overfitting, and intra-class facial appearance variations. FLEPNet, a texture-based feature-level ensemble parallel network for FER, is proposed in this study and proved to solve the aforementioned problems. Our parallel network FLEPNet uses multi-scale convolutional and multi-scale residual block-based DCNN as building blocks. First, we consider modified homomorphic filtering to normalize the illumination effectively, which minimizes the intra-class difference. The deep networks are then protected against having insufficient training data by using texture analysis on face expression images to identify multiple attributes. Four texture features are extracted and combined with the image's original characteristics. Finally, the integrated features retrieved by two networks are used to classify seven facial expressions. Experimental results reveal that the proposed technique achieves an average accuracy of 0.9914, 0.9894, 0.9796, 0.8756, and 0.8072 on Japanese Female Facial Expressions, Extended CohnKanade, Karolinska Directed Emotional Faces, Real-world Affective Face Database, and Facial Expression Recognition 2013 databases, respectively. Moreover, experimental outcomes depict significant reliability when compared to competing approaches.

Quaternion Capsule Neural Network With Region Attention for Facial Expression Recognition in Color Images

Facial expression recognition (FER) is an essential subject of computer vision and human-computer interaction. It has been reported that many factors are closely related to the FER performance such as the pose, facial muscle variations, and the ignored color information in facial images. In this study, we propose a quaternion capsule neural network (Q-CapsNet) with region attention mechanism for FER in color images. The proposed Q-CapsNet is an end-to-end deep learning framework, which adopts the concept of quaternion theory to the Capsule Neural Network and further uses an attention mechanism to focus on the facial region of interests (ROIs). The Q-CapsNet addresses the FER problems in the following aspects. First, the internal dependencies between color channels are captured by the quaternion technique, which is of great importance in color image processing. Second, the Q-CapsNet could disentangle latent geometry information from facial images by quaternion capsules and quaternion routing algorithm. Third, inspired by the fact that facial expressions are mainly determined by several vital facial regions, the region attention mechanism is introduced to extract emotional features from the ROIs of the face. The proposed Q-CapsNet is evaluated on four public color FER datasets including MMI, Oulu-CASIA, RAF-DB, and SFEW. The comprehensive experiments and visualization results demonstrate that the proposed network outperforms comparative models and the state-of-the-art FER methods.

Impact of Activation, Optimization, and Regularization Methods on the Facial Expression Model Using CNN.

When it comes to conveying sentiments and thoughts, facial expressions are quite effective. For human-computer collaboration, data-driven animation, and communication between humans and robots to be successful, the capacity to recognize emotional states in facial expressions must be developed and implemented. Recently published studies have found that deep learning is becoming increasingly popular in the field of image categorization. As a result, to resolve the problem of facial expression recognition (FER) using convolutional neural networks (CNN), increasingly substantial efforts have been made in recent years. Facial expressions may be acquired from databases like CK+ and JAFFE using this novel FER technique based on activations, optimizations, and regularization parameters. The model recognized emotions such as happiness, sadness, surprise, fear, anger, disgust, and neutrality. The performance of the model was evaluated using a variety of methodologies, including activation, optimization, and regularization, as well as other hyperparameters, as detailed in this study. In experiments, the FER technique may be used to recognize emotions with an Adam, Softmax, and Dropout Ratio of 0.1 to 0.2 when combined with other techniques. It also outperforms current FER techniques that rely on handcrafted features and only one channel, as well as has superior network performance compared to the present state-of-the-art techniques.

A Method of Improving Accuracy in Expression Recognition

In order to improve the accuracy of a special kind of facial expression recognition problem, a method for precise face detection and segmentation combined with the particle swarm optimization is proposed. The method uses three key technologies: skin color segmentation, particle swarm search and curve approximation. Firstly, the face contour is roughly obtained through skin color segmentation. Secondly, the accurate face position is detected by particle swarm optimization. Thirdly, the face contour is reduced and regulated further via the curve approximation. The experimental results show that this method can eliminate the interference factor, and then improve the accuracy of expression recognition.

O reconhecimento de expressões faciais de emoção e os problemas interpessoais diferem entre alcoolistas em diferentes estágios de abstinência?

O reconhecimento de expressões faciais de emoção e os problemas interpessoais diferem entre alcoolistas em diferentes estágios de abstinência?

4D Facial Expression Recognition Using Geometric Landmark-based Axes-angle Feature Extraction

The primary goal of this paper is to describe a proposed framework for identifying human face expressions. A methodology has been proposed and developed to identify facial emotions using an axes-angular feature extracted from facial landmarks for 4D dynamic facial expression video data. The 4D facial expression recognition (FER) problem is modeled as an unbalanced problem using the full video sequence. The proposed dataset includes landmarks that are positioned to be fiducial features: around the brows, eyes, nose, cheeks, and lips. Following the initial facial landmark preprocessing, feature extraction is carried out. Input feature vectors from gamma axes and magnitudes in three-dimensional Euclidean space are constructed. This paper develops a new feature vector by concatenating the angle and magnitude features and compares them to a model created independently using angle features. In all three models, several filter-based feature selection techniques are used to estimate feature importance. The framework employs a multi-class support vector machine (SVM) on MATLAB, and the true positive rate (TPR) and average recognition rates (ARR) are used as performance metrics. In terms of average classification accuracy, the final results are compared to conventional state-of-the-art approaches. The results revealed a highly informative combined feature, demonstrating the efficiency of the proposed landmark-based strategy to classify facial expressions.

A Facial Expressions Recognition Method Using Residual Network Architecture for Online Learning Evaluation

Facial expression recognition (FER) has been widely researched in recent years, with successful applications in a range of domains such as monitoring and warning of drivers for safety, surveillance, and recording customer satisfaction. However, FER is still challenging due to the diversity of people with the same facial expressions. Currently, researchers mainly approach this problem based on convolutional neural networks (CNN) in combination with architectures such as AlexNet, VGGNet, GoogleNet, ResNet, SENet. Although the FER results of these models are getting better day by day due to the constant evolution of these architectures, there is still room for improvement, especially in practical applications. In this study, we propose a CNN-based model using a residual network architecture for FER problems. We also augment images to create a diversity of training data to improve the recognition results of the model and avoid overfitting. Utilizing this model, this study proposes an integrated system for learning management systems to identify students and evaluate online learning processes. We run experiments on different datasets that have been published for research: CK+, Oulu-CASIA, JAFFE, and collected images from our students (FERS21). Our experimental results indicate that the proposed model performs FER with a significantly higher accuracy compared with other existing methods.

Transfer Model Collaborating Metric Learning and Dictionary Learning for Cross-Domain Facial Expression Recognition

Facial expression recognition has drawn increasing attention because of its great potential in human behavior analysis. Traditional recognition models usually assume that the training set is sufficient, and the training and testing data sets have the same distribution. However, these two factors are not satisfied in some cases. In this study, a transfer model collaborating metric learning and dictionary learning called TMMLDL is proposed to address the transfer facial expression recognition problem. To reduce the impact of cross-domain distribution variation, the information of global structure and pairwise constraints are utilized among training images in different domains. In particular, a discriminative metric space is learned into dictionary learning procedure such that the dictionary items can well present the discriminative information of different facial expression classes in the metric subspace. The proposed model tunes dictionary and metric space in an alternative optimization algorithm, which is guaranteed to obtain the optimal model parameters simultaneously. The experimental results on nine cross-domain facial expression classification tasks show that the proposed model achieves the satisfactory recognition performance.

DEEP LEARNING TECHNOLOGY OF CONVOLUTIONALNEURAL NETWORKS FOR FACIAL EXPRESSION RECOGNITION

The application of deep learning convolutional neural networks for solving the problem of automated facial expression recognition and determination of emotions of a person is analyzed. It is proposed to use the advantages of the transfer approach to deep learning convolutional neural networks training to solve the problem of insufficient data volume in sets of images with different facial expressions. Most of these datasets are labeled in accordance with a facial coding system based on the units ofhuman facial movement. The developed technology of transfer learning of the public deep learning convolutional neural networksfamilies DenseNet and MobileNet, with the subsequent “fine tuning”of the network parameters, allowed to reduce the training time and computational resources when solving the problem of facial expression recognition without losing the reliability of recognition of motor units. During the development of deep learning technology for convolutional neural networks, the following tasks were solved. Firstly, the choice of publicly available convolutional neural networks of the DenseNet and MobileNet families pre-trained on the ImageNet dataset was substantiated, taking into account the peculiarities of transfer learning for the task of recognizing facial expressions and determining emotions. Secondary, amodel of a deep convolutional neural network and a method for its training have been developed for solving problems of recognizing facial expressions and determining human emotions, taking into account the specifics of the selected pretrained convolutional neural networks. Thirdly, the developed deep learning technology was tested,and finally, the resource intensity and reliability of recognition of motor units on the DISFA set were assessed. The proposed technology of deep learning of convolutional neural networks can be used in the developmentof systems for automatic recognition of facial expressions and determination of humanemotions for both stationary and mobile devices. Further modification of the systems for recognizing motor units of human facial activity in order to increase the reliability of recognition is possible using of theaugmentation technique.

On the pitfalls of learning with limited data: A facial expression recognition case study

Deep learning models need large amounts of data for training. In video recognition and classification, significant advances were achieved with the introduction of new large databases. However, the creation of large-databases for training is infeasible in several scenarios. Thus, existing or small collected databases are typically joined and amplified to train these models. Nevertheless, training neural networks on limited data is not straightforward and comes with a set of problems. In this paper, we explore the effects of stacking databases, model initialization, and data amplification techniques when training with limited data on deep learning models’ performance. We focused on the problem of Facial Expression Recognition from videos. We performed an extensive study with four databases at a different complexity and nine deep-learning architectures for video classification. We found that (i) complex training sets translate better to more stable test sets when trained with transfer learning and synthetically generated data, but their performance yields a high variance; (ii) training with more detailed data translates to more stable performance on novel scenarios (albeit with lower performance); (iii) merging heterogeneous data is not a straightforward improvement, as the type of augmentation and initialization is crucial; (iv) classical data augmentation cannot fill the holes created by joining largely separated datasets; and (v) inductive biases help to bridge the gap when paired with synthetic data, but this data is not enough when working with standard initialization techniques.

Facial expression recognition on a quantum computer

We address the problem of facial expression recognition and show a possible solution using a quantum machine learning approach. In order to define an efficient classifier for a given dataset, our approach substantially exploits quantum interference. By representing face expressions via graphs, we define a classifier as a quantum circuit that manipulates the graphs adjacency matrices encoded into the amplitudes of some appropriately defined quantum states. We discuss the accuracy of the quantum classifier evaluated on the quantum simulator available on the IBM Quantum Experience cloud platform, and compare it with the accuracy of one of the best classical classifier.

Facial Expression Recognition Based on Multi-Features Cooperative Deep Convolutional Network

This paper addresses the problem of Facial Expression Recognition (FER), focusing on unobvious facial movements. Traditional methods often cause overfitting problems or incomplete information due to insufficient data and manual selection of features. Instead, our proposed network, which is called the Multi-features Cooperative Deep Convolutional Network (MC-DCN), maintains focus on the overall feature of the face and the trend of key parts. The processing of video data is the first stage. The method of ensemble of regression trees (ERT) is used to obtain the overall contour of the face. Then, the attention model is used to pick up the parts of face that are more susceptible to expressions. Under the combined effect of these two methods, the image which can be called a local feature map is obtained. After that, the video data are sent to MC-DCN, containing parallel sub-networks. While the overall spatiotemporal characteristics of facial expressions are obtained through the sequence of images, the selection of keys parts can better learn the changes in facial expressions brought about by subtle facial movements. By combining local features and global features, the proposed method can acquire more information, leading to better performance. The experimental results show that MC-DCN can achieve recognition rates of 95%, 78.6% and 78.3% on the three datasets SAVEE, MMI, and edited GEMEP, respectively.