Facial Expression Recognition Accuracy Research Articles

Parameter optimization in convolutional neural network for improving performance of facial expression recognition

The recognition of similar facial expressions presents a notable challenge, necessitating a focus on the parameters within the fundamental Convolutional Neural Network (CNN) architecture, which serves as a cornerstone in the field of image classification. This research endeavor aims to enhance the model's capacity for facial expression recognition by employing a controlled variable method to examine two specific parameters in a self-designed small CNN: the number of filters and convolutional layers. More specifically, while the filters were fixed at 3, the layers varied from 3 to 6 to 9. Similarly, as the number of the filters rose to 6, the number of the layers also incremented from 3 to 6 to 9. Furthermore, while the number of the filters reached 12, the number of the layers went from 3 to 6 to 9 too. Finally, with the filters increasing to 24, the layers rose from 3 to 6 to 9 as well. Experimental results indicate that both increasing the number of filters and convolutional layers can increment the performance of model in facial expression recognition. Furthermore, increasing the number of filters can exert a more prominent influence on improving the accuracy of facial expression recognition.

Factors Associated with Children’s Humane Attitudes toward Animals

Abstract Various studies have suggested a range of different factors associated with children’s attitudes toward animals; however, in Chile, it is an area that has not been studied. The current study administered the Intermediate Attitude Scale (IAS, which measures attitudes toward the humane treatment of animals) to 286 children in third and fourth grade and analyzed its association with the following variables: type of school, grade, gender, current pet ownership, the recognition accuracy of dog facial expressions, and human-directed empathy. The association between the IAS and potential factors was measured through linear mixed models. Children with higher human-directed empathy scores, fourth-grade children, and children who were more accurate in recognizing the happiness emotion on the dog’s face were associated with more positive attitudes toward the humane treatment of animals. These findings contribute to the growing literature regarding the importance of positive attitudes toward animals and their connection to human-directed empathy.

Facial expression recognition based on adaptation of the classifier to videos of the user

In this paper, an approach that can significantly increase the accuracy of facial emotion recognition by adapting the model to the emotions of a particular user (e.g., smartphone owner) is considered. At the first stage, a neural network model, which was previously trained to recognize facial expressions in static photos, is used to extract visual features of faces in each frame. Next, the face features of video frames are aggregated into a single descriptor for a short video fragment. After that a neural network classifier is trained. At the second stage, it is proposed that adaptation (fine-tuning) to this classifier should be performed using a small set of video data with the facial expressions of a particular user. After emotion classification, the user can adjust the predicted emotions to further improve the accuracy of a personal model. As part of an experimental study for the RAVDESS dataset, it has been shown that the approach with model adaptation to a specific user can significantly (up to 20 – 50 %) improve the accuracy of facial expression recognition in the video.

Analyzing the Effect of Diverse Gaze and Head Direction on Facial Expression Recognition With Photo-Reflective Sensors Embedded in a Head-Mounted Display.

As one of the facial expression recognition techniques for Head-Mounted Display (HMD) users, embedded photo-reflective sensors have been used. In this paper, we investigate how gaze and face directions affect facial expression recognition using the embedded photo-reflective sensors. First, we collected a dataset of five facial expressions (Neutral, Happy, Angry, Sad, Surprised) while looking in diverse directions by moving 1) the eyes and 2) the head. Using the dataset, we analyzed the effect of gaze and face directions by constructing facial expression classifiers in five ways and evaluating the classification accuracy of each classifier. The results revealed that the single classifier that learned the data for all gaze points achieved the highest classification performance. Then, we investigated which facial part was affected by the gaze and face direction. The results showed that the gaze directions affected the upper facial parts, while the face directions affected the lower facial parts. In addition, by removing the bias of facial expression reproducibility, we investigated the pure effect of gaze and face directions in three conditions. The results showed that, in terms of gaze direction, building classifiers for each direction significantly improved the classification accuracy. However, in terms of face directions, there were slight differences between the classifier conditions. Our experimental results implied that multiple classifiers corresponding to multiple gaze and face directions improved facial expression recognition accuracy, but collecting the data of the vertical movement of gaze and face is a practical solution to improving facial expression recognition accuracy.

Lightweight Facial Expression Recognition Based on Class-Rebalancing Fusion Cumulative Learning

In the research of Facial Expression Recognition (FER), the inter-class of facial expression data is not evenly distributed, the features extracted by networks are insufficient, and the FER accuracy and speed are relatively low for practical applications. Therefore, a lightweight and efficient method based on class-rebalancing fusion cumulative learning for FER is proposed in our research. A dual-branch network (Regular feature learning and Rebalancing-Cumulative learning Network, RLR-CNet) is proposed, where the RLR-CNet uses the improvement in the lightweight ShuffleNet with two branches (feature learning and class-rebalancing) based on cumulative learning, which improves the efficiency of our model recognition. Then, to enhance the generalizability of our model and pursue better recognition efficiency in real scenes, a random masking method is improved to process datasets. Finally, in order to extract local detailed features and further improve FER efficiency, a shuffle attention module (SA) is embedded in the model. The results demonstrate that the recognition accuracy of our RLR-CNet is 71.14%, 98.04%, and 87.93% on FER2013, CK+, and RAF-DB, respectively. Compared with other FER methods, our method has great recognition accuracy, and the number of parameters is only 1.02 MB, which is 17.74% lower than that in the original ShuffleNet.

Trusted emotion recognition based on multiple signals captured from video

Emotion recognition based on facial expressions has low accuracy and doubtful reliability because of the existence of fake expressions. In this paper, a method is proposed to recognize fake emotions based on multivisual information generated from single information sources, including facial expressions, eye states and physiological signals captured from video. An algorithm based on a graph neural network is used to extract spatial and spectral domain features from facial images for facial expression recognition. A model-based method is used for decomposing RGB signals into heart rates. A deep model trained by a labeled dataset that we created is used to segment the eye region. After obtaining the signals extracted from video, different fusion strategies are applied to evaluate emotion recognition performance based on multiple signals. In the experiment, the CK+, TFEID, JAFFE, RAF, PURE, and ESLD datasets are used to measure the accuracy of facial expression recognition, heart rate detection and eye region segmentation. The results show that multimodality is effective in improving accuracy and that eye state can be considered a cue for trusted emotion recognition. Compared with the method based on the eye state and noncontact physiological signal, the accuracy of multimodality can be improved by 26.19% and 9.52%, respectively.

Effect of low luminance on face recognition in adults with central and peripheral vision loss.

To investigate the effect of low luminance on face recognition, specifically facial identity discrimination (FID) and facial expression recognition (FER), in adults with central vision loss (CVL) and peripheral vision loss (PVL) and to explore the association between clinical vision measures and low luminance FID and FER. Participants included 33 adults with CVL, 17 with PVL and 20 controls. FID and FER were assessed under photopic and low luminance conditions. For the FID task, 12 sets of three faces with neutral expressions were presented and participants asked to indicate the odd-face-out. For FER, 12 single faces were presented and participants asked to name the expression (neutral, happy or angry). Photopic and low luminance visual acuity (VA) and contrast sensitivity (CS) were recorded for all participants and for the PVL group, Humphrey Field Analyzer (HFA) 24-2 mean deviation (MD). FID accuracy in CVL, and to a lesser extent PVL, was reduced under low compared with photopic luminance (mean reduction 20% and 8% respectively; p < 0.001). FER accuracy was reduced only in CVL (mean reduction 25%; p < 0.001). For both CVL and PVL, low luminance and photopic VA and CS were moderately to strongly correlated with low luminance FID (ρ = 0.61-0.77, p < 0.05). For PVL, better eye HFA 24-2 MD was moderately correlated with low luminance FID (ρ = 0.54, p = 0.02). Results were similar for low luminance FER. Together, photopic VA and CS explained 75% of the variance in low luminance FID, and photopic VA explained 61% of the variance in low luminance FER. Low luminance vision measures explained little additional variance. Low luminance significantly reduced face recognition, particularly for adults with CVL. Worse VA and CS were associated with reduced face recognition. Clinically, photopic VA is a good predictor of face recognition under low luminance conditions.

The Performance Analysis of Facial Expression Recognition System Using Local Regions and Features

Different parts of our face contribute to overall facial expressions, such as anger, happiness and sadness in distinct ways. This paper investigates the degree of importance of different human face parts to the accuracy of Facial Expression Recognition (FER). In the context of machine learning, FER refers to a problem where a computer vision system is trained to automatically detect the facial expression from a presented facial image. This is a difficult image classification problem that is not yet fully solved and has received significant attention in recent years, mainly due to the increased number of possible applications in daily life. To establish the extent to which different human face parts contribute to overall facial expression, various sections have been extracted from a set of facial images and then used as inputs into three different FER systems. In terms of the recognition rates for each facial section, this result confirms that various regions of the face have different levels of importance regarding the accuracy rate achieved by an associated FER system.

Facial expression recognition network with slow convolution and zero-parameter attention mechanism

Facial expression recognition (FER) has been successfully applied to the fields of automatic driving, classroom education, and many other industries. However, a lot of existing problems induced by different lighting, posture, and skin color make many difficulties in extracting effective features, resulting in low accuracy of FER. to address these issues, a novel slow evolution and zero parameter attention mechanism (SC-ZAM) is proposed to reliably extract discriminative feature from facial images for FER. In the proposed SC-ZAM, Slow convolution module is used to reduce the speed of image dimensionality reduction, extract considerably reliable facial features, and enhances the representation of these features to extract more effective global information of facial expressions. Meanwhile, zero-parameter attention modules are applied to enhance the focus on the local discriminative feature areas. The improved attention module focuses on highly distinguishable facial features and restrains the weight of non-facial areas to locate more detailed local facial areas in the global information extracted from the network. Experimental results demonstrate that our SC-ZAM method is robust against different FER tasks on our infant dataset and several public datasets, and is superior to other state-of-the-art methods.

Typical Facial Expression Network Using a Facial Feature Decoupler and Spatial-Temporal Learning

Facial expression recognition (FER) accuracy is often affected by an individuals unique facial characteristics. Recognition performance can be improved if the influence from these physical characteristics is minimized. Using video instead of single image for FER provides better results but requires extracting temporal features and the spatial structure of facial expressions in an integrated manner. We propose a new network called Typical Facial Expression Network (TFEN) to address both challenges. TFEN uses two deep two-dimensional (2D) convolutional neural networks (CNNs) to extract facial and expression features from input video. A facial feature decoupler decouples facial features from expression features to minimize the influence from inter-subject face variations. These networks combine with a 3D CNN and form a spatial-temporal learning network to jointly explore the spatial-temporal features in a video. A facial recognition network works as an adversarial network to refine the facial feature decoupler and the network performance by minimizing the residual influence of facial features after decoupling. The whole network is trained with an adversarial algorithm to improve FER performance. TFEN was evaluated on four popular dynamic FER datasets. Experimental results show TFEN achieves or outperforms the recognition accuracy of state-of-the-art approaches.

Geometry-Aware Facial Expression Recognition via Attentive Graph Convolutional Networks

Learning discriminative representations with good robustness from facial observations serves as a fundamental step towards intelligent facial expression recognition (FER). In this paper, we propose a novel geometry-aware FER framework to boost the FER performance based on both the geometric and appearance knowledge. Specifically, we propose an encoding strategy for facial landmarks, and adopt a graph convolutional network (GCN) to fully explore the structural information of the facial components behind different expressions. A convolutional neural network (CNN) is further applied to the whole facial observation to learn the global characteristics of different expressions. The features from these two networks are fused into a comprehensive high-semantic representation, which promotes the FER reasoning from both visual and structural perspectives. Moreover, to facilitate the networks to concentrate on the most informative facial regions and components, we introduce multi-level attention mechanisms into the proposed framework, which enhance the reliability of the learned representations for effective FER. Experiments on two challenging FER benchmarks demonstrate that the attentive graph-based learning on the facial geometry boosts the FER accuracy. Furthermore, the insensitivity of the geometric information to the appearance variations also improves the generalization of the proposed framework.

Multi-Class Facial Emotion Recognition Using Hybrid Dense Squeeze Network

Automatic facial expression recognition (FER) is utilized in various applications like psychoanalysis, intelligent driving, robot manufacturing, etc. Numerous researchers have been looking for better techniques to improve the accuracy of FER. In fact, FER under laboratory conditions has almost achieved top accuracy. Besides, label deviations or errors caused by annotators’ subjectivity also make the FER task much tougher. Thus, more and more researchers begin to find new ways to handle with the FER problems. In this work, a new deep learning (DL) model called dense squeeze network with improved red deer optimization (DenseSNet_IRDO) is proposed for the recognition of facial emotions. The steps used for FER are pre-processing, fused deep feature extraction-selection and classification. Initially, the facial images are pre-processed using improved trilateral filter (ITF) for improving the quality of images. Next, the fusion of feature extraction and selection is performed using the DenseSNet. Here the extraction of deep features is done with the dense network and the relevant features are selected with the squeeze network. Finally, the last layer of squeeze network performs the classification of various facial emotions. Here, the loss in the classification is optimized using IRDO. This DenseSNet_IRDO architecture is more robust and avoids overfitting that occurs while training the small dataset. The datasets used in this work are CK[Formula: see text], JAFEE and FERFIN. The proposed FER classification using datasets CK[Formula: see text], JAFEE and FERFIN with DenseSNet_IRDO model achieved the accuracy of 99.91%, 99.90% and 99.89%, respectively. Thus, the proposed DenseSNet_IRDO classifier model obtained higher accuracy in the detection of FER than other methods.

SqueezExpNet: Dual-stage convolutional neural network for accurate facial expression recognition with attention mechanism

Facial expression recognition (FER) using a deep convolutional neural network (DCNN) is important and challenging. Although a substantial effort is made to increase FER accuracy through DCNN, previous studies are still not sufficiently generalisable for real-world applications. Traditional FER studies are mainly limited to controlled lab-posed frontal facial images, which lack the challenges of motion blur, head poses, occlusions, face deformations and lighting under uncontrolled conditions. In this work, we proposed a SqueezExpNet architecture that can take advantage of local and global facial information for a highly accurate FER system that can handle environmental variations. Our network was divided into two stages: a geometrical attention stage that possesses a SqueezeNet-like architecture to obtain local highlight information and a spatial texture stage comprising several squeezed and expanded layers to exploit high-level global features. In particular, we created a weighted mask of 3D face landmarks and used element-wise multiplication with a spatial feature in the first stage to draw attention to important local facial regions. Next, we input the face spatial image and its augmentations into the second stage of the network. Finally, like a classifier, a recurrent neural network was designed to collaborate the highlighted information from dual stages rather than simply using the SoftMax function, thereby aiding in overcoming the uncertainties. Experiments covering basic and compound FER tasks were performed using the three leading facial expression datasets. Our strategy outperformed the existing DCNN methods and achieved state-of-the-art results. The developed architecture, adopted research methodology and reported findings may find potential applications of real-time FER in surveillance, health and feedback systems.

Facial Emotion Recognition

Abstract: Facial Emotion Recognition plays a significant role in interacting with computers which help us in various fields like medical processes, to present content on the basis of human mood, security and other fields. It is challenging because of heterogeneity in human faces, lighting, orientation, poses and noises. This paper aims to improve the accuracy of facial expression recognition. There has been much research done on the fer2013 dataset using CNN (Convolution Neural Network) and their results are quite impressive. In this work we performed CNN on the fer2013 dataset by adding images to improve the accuracy. To our best knowledge, our model achieves the accuracy of 70.23 % on fer2013 dataset after adding images in training and testing parts of disgusted class.

Expression Recognition Based on Multi-level Multi-model Fusion Deep Convolutional Neural Network

In the research of Facial Expression Recognition (FER), training networks using deep learning methods often relies on data sets with accurate label and balanced number of each classification. However, when training on low-quality data sets, the performance of traditional models is insufficient. In order to solve the problem, the paper proposes an expression recognition method based on Multi-level Multi-model Fusion Deep Convolutional Neural Network. The Deep Convolutional Neural Network is composed of three models, ResNet-50, ResNet-101 and InceptionNet-V3, which have similar performance on ImageNet. We use the weighted average fusion algorithm as our fusion method. According to the accuracy of the three models, weighted fusion is carried out, and the model with high accuracy is given higher weight. It increases the width of the network and the number of parameters. At the same time, it can solve the problem of accuracy degradation when the number of layers in the deep network increases. We choose FER2013, a widely recognized low-quality data set, for the experiment. The experimental results based on FER2013 data set show that the facial expression recognition accuracy of this method reach 71.78%, which is significantly better than ResNet-50, ResNet-101 and InceptionNet-V3. Compared with other models, it also verifies the effectiveness of the proposed model in identifying low-quality data sets.

Facial expression recognition based on regional adaptive correlation

AbstractTo address the problem that the features extracted by CNN‐based facial expression recognition (FER) do not consider structural information, a region adaptive correlation deep network (RACN) is proposed. The network consists of two branches. In one branch, the features obtained by applying CNN to facial sub‐blocks are used as the input of the proposed second‐order region correlation network (SRCN), which obtains structural features by adaptively learning the correlation of facial regions. Furthermore, they are fused with the parallel branch‐extracted global features to obtain a comprehensive high‐semantic feature representation. Finally, weights are assigned to the two features through the channel attention mechanism for more accurate expression classification. Experimental results show that our method can effectively extract expression features in an end‐to‐end manner, improve the accuracy of FER, and achieve competitive performance without relying on any a priori knowledge. And the region‐adaptive correlation feature extraction branch RACN can be applied to other deep learning networks to extract discriminative structural‐adaptive features. To the best of our knowledge, our work is the first to enrich the feature representation for end‐to‐end static FER by adaptively obtaining more discriminative regional adaptive correlation feature vectors via the autocorrelation matrix combined with CNN compared to the existing literature.

Insomnia and impacts on facial expression recognition accuracy, intensity and speed: A meta-analysis

Facial expressions provide nonverbal cues that are important for delivering and interpreting human emotions. Previous studies have shown that the ability to interpret facial emotions correctly could be partially impaired in sleep-deprived people. People with insomnia might also suffer from sleep loss, so we assumed that facial expression recognition ability might also be impaired in people with insomnia. Despite a growing body of research exploring insomnia's potential impacts on facial expression recognition, conflicting results have been reported, and no systematic review of this research body has been conducted. In this study, after screening 1100 records identified through database searches, six articles examining insomnia and facial expression recognition ability were included in a quantitative synthesis. The main outcomes were classification accuracy (ACC), reaction time (RT), and intensity rating—the three most studied facial expression processing variables. Subgroup analysis was performed to identify altered perceptions according to the facial expressions of four emotions—happiness, sadness, fear, and anger—used to examine insomnia and emotion recognition. The pooled standard mean differences (SMDs) and corresponding 95% confidence intervals (CIs) demonstrated that facial expression recognition among people with insomnia was less accurate (SMD = −0.30; 95% CI: −0.46, −0.14) and slower (SMD = 0.67; 95% CI: 0.18, −1.15) compared to good sleepers. The classification ACC of fearful expression was lower in the insomnia group (SMD = −0.66; 95% CI: −1.02, −0.30). This meta-analysis was registered using PROSPERO.

A Method for Analyzing Learning Sentiment Based on Classroom Time‐Series Images

With the development of smart classrooms, analyzing students’ emotions for classroom learning is an effective means of accurately capturing their learning process. Although facial expression‐based emotion analysis methods are effective in analyzing classroom learning emotions, current research focuses on facial expressions and does not consider the fact that expressions in different postures do not represent the same emotions. To provide a continuous and deeper understanding of students’ learning emotions, this study proposes an algorithm to characterize learning emotions based on classroom time‐series image data. First, face expression data for classroom scenarios are established to address the lack of expression databases in real teaching environments. Second, to improve the accuracy of facial expression recognition, a residual channel cross transformer masking net expression recognition model is proposed in this paper. Finally, to address the problem that the existing research dimension of learning emotion is too single, this paper uses the facial expression and head posture data obtained from deep learning models for fusion analysis and innovatively proposes a Dempster–Shafer evidence‐theoretic fusion model to characterize the learning emotion within the lecture duration of knowledge points. The experiments show that both the proposed expression recognition model and the learning sentiment analysis algorithm have good performance, with the expression recognition model achieving an accuracy of 73.58% on the FER2013 dataset. The proposed learning emotion analysis method provides technical support for holistic analysis of student learning effects and evaluation of students’ level of understanding of the knowledge points.

Hybrid Convolutional Neural Network and Long Short-Term Memory Approach for Facial Expression Recognition

Facial Expression Recognition (FER) has been an important field of research for several decades. Extraction of emotional characteristics is crucial to FERs, but is complex to process as they have significant intra-class variances. Facial characteristics have not been completely explored in static pictures. Previous studies used Convolution Neural Networks (CNNs) based on transfer learning and hyperparameter optimizations for static facial emotional recognitions. Particle Swarm Optimizations (PSOs) have also been used for tuning hyperparameters. However, these methods achieve about 92 percent in terms of accuracy. The existing algorithms have issues with FER accuracy and precision. Hence, the overall FER performance is degraded significantly. To address this issue, this work proposes a combination of CNNs and Long Short-Term Memories (LSTMs) called the HCNN-LSTMs (Hybrid CNNs and LSTMs) approach for FERs. The work is evaluated on the benchmark dataset, Facial Expression Recog Image Ver (FERC). Viola-Jones (VJ) algorithms recognize faces from preprocessed images followed by HCNN-LSTMs feature extractions and FER classifications. Further, the success rate of Deep Learning Techniques (DLTs) has increased with hyperparameter tunings like epochs, batch sizes, initial learning rates, regularization parameters, shuffling types, and momentum. This proposed work uses Improved Weight based Whale Optimization Algorithms (IWWOAs) to select near-optimal settings for these parameters using best fitness values. The experimental findings demonstrated that the proposed HCNN-LSTMs system outperforms the existing methods.

Combining Deep Convolutional Neural Networks With Stochastic Ensemble Weight Optimization for Facial Expression Recognition in the Wild

Although recent emotion recognition methods (based on facial expression cues) achieve excellent performance in controlled scenarios, the recognition of emotion in the wild remains a challenging problem because of occlusion, large head poses, illumination variations, etc. Recent advances in deep learning show that combining an ensemble of deep learning models can considerably outperform the approach of using only a single deep learning model for challenging recognition problems. This paper presents a novel ensemble deep learning method, “deep convolutional neural network (DCNN) ensemble classifier”, for improved facial expression recognition (FER) in the wild. Our proposed DCNN ensemble classifier is novel in terms of the following aspects: (1) the process of finding ensemble weights for combining DCNN decision outputs is formulated as a stochastic optimization problem (via simulated annealing) in which the energy to be minimized represents the generalized (test) classification error of the DCNN ensemble and (2) for the creation of DCNN ensemble members, we propose the combined use of different types of face representations and bagging (T. G. Dietterich, 2000), which is quite useful in increasing the diversity of the DCNN ensemble. Extensive and comparative experiments on three wild FER datasets, namely FER2013, SFEW2.0, and RAF-DB, show that the proposed DCNN ensemble classifier achieves competitive FER performances when compared with other recently developed methods—76.69%, 58.68%, and 87.13% of FER accuracy under the FER2013, SFEW2.0, and RAF-DB evaluation protocols, respectively.