FER2013 Dataset Research Articles

Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity

Emotion recognition is defined as identifying human emotion and is directly related to different fields such as human–computer interfaces, human emotional processing, irrational analysis, medical diagnostics, data-driven animation, human–robot communication, and many more. This paper proposes a new facial emotional recognition model using a convolutional neural network. Our proposed model, “ConvNet”, detects seven specific emotions from image data including anger, disgust, fear, happiness, neutrality, sadness, and surprise. The features extracted by the Local Binary Pattern (LBP), region based Oriented FAST and rotated BRIEF (ORB) and Convolutional Neural network (CNN) from facial expressions images were fused to develop the classification model through training by our proposed CNN model (ConvNet). Our method can converge quickly and achieves good performance which the authors can develop a real-time schema that can easily fit the model and sense emotions. Furthermore, this study focuses on the mental or emotional stuff of a man or woman using the behavioral aspects. To complete the training of the CNN network model, we use the FER2013 databases at first, and then apply the generalization techniques to the JAFFE and CK+ datasets respectively in the testing stage to evaluate the performance of the model. In the generalization approach on the JAFFE dataset, we get a 92.05% accuracy, while on the CK+ dataset, we acquire a 98.13% accuracy which achieve the best performance among existing methods. We also test the system’s success by identifying facial expressions in real-time. ConvNet consists of four layers of convolution together with two fully connected layers. The experimental results show that the ConvNet is able to achieve 96% training accuracy which is much better than current existing models. However, when compared to other validation methods, the suggested technique was more accurate. ConvNet also achieved validation accuracy of 91.01% for the FER2013 dataset. We also made all the materials publicly accessible for the research community at: https://github.com/Tanoy004/Emotion-recognition-through-CNN.

Mood Based Music Playlist Generator Using Convolutional Neural Network

Abstract: Music is used in one’s everyday life to modulate, enhance, and plummet undesirable emotional states like stress, fatigue, or anxiety. Even though in today's modern world, with the ever-increasing advancement in the area of multimedia content and usage, various intuitive music players with various options have been developed, the user still has to manually browse through the list of songs and select songs that complement and suit his or her current mood. This paper aims to solve this issue of the manual finding of songs to suit one’s mood along with creating a high accuracy CNN model for Facial emotion recognition. Through the webcam, the emotional state can be deduced from facial expressions. To create a neural network model, the CNN classifier was used. This model is trained and tested using OpenCV to detect mood from facial expressions. A playlist will be generated through the system according to the mood predicted by the model. The model used for predicting the mood obtained an accuracy of 97.42% with 0.09 loss on the training data from the FER2013 Dataset. Keywords: Convolutional neural networks, Facial emotion recognition, FER2013, Music, OpenCV.

Expression Recognition Using Improved AlexNet Network in Robot Intelligent Interactive System

Aiming at the insufficient feature extraction in the expression feature extraction stage of traditional convolutional neural network and the misclassification of mislabeled samples, an expression recognition and robot intelligent interaction method using deep learning is proposed. First, in image preprocessing, the dimension of the color image is reduced by image gray adjustment to reduce the amount of calculation, the shadow interference is eliminated by the average method, and the image is enhanced by histogram equalization. Second, multichannel convolution is used to replace the single convolution size in the second convolution layer in AlexNet, the Global Average Pooling layer is introduced to replace the fully connected layer, and Batch Normalization is introduced to improve the feature extraction ability of the model and avoid gradient explosion. Finally, the Focal Loss is improved by setting the probability threshold to avoid the impact of mislabeling samples on the classification performance of the model. The experimental results show that the recognition accuracy of the model on FER2013 data set is 98.36%. The effectiveness of the algorithm is verified on the intelligent interactive system of service robot based on expression recognition. Compared with other expression recognition methods, the proposed method can extract more expression features and recognize facial expression more accurately.

Precision-Based Weighted Blending Distributed Ensemble Model for Emotion Classification

Focusing on emotion recognition, this paper addresses the task of emotion classification and its performance with respect to accuracy, by investigating the capabilities of a distributed ensemble model using precision-based weighted blending. Research on emotion recognition and classification refers to the detection of an individual’s emotional state by considering various types of data as input features, such as textual data, facial expressions, vocal, gesture and physiological signal recognition, electrocardiogram (ECG) and electrodermography (EDG)/galvanic skin response (GSR). The extraction of effective emotional features from different types of input data, as well as the analysis of large volume of real-time data, have become increasingly important tasks in order to perform accurate classification. Taking into consideration the volume and variety of the examined problem, a machine learning model that works in a distributed manner is essential. In this direction, we propose a precision-based weighted blending distributed ensemble model for emotion classification. The suggested ensemble model can work well in a distributed manner using the concepts of Spark’s resilient distributed datasets, which provide quick in-memory processing capabilities and also perform iterative computations effectively. Regarding model validation set, weights are assigned to different classifiers in the ensemble model, based on their precision value. Each weight determines the importance of the respective classifier in terms of its performing prediction, while a new model is built upon the derived weights. The produced model performs the task of final prediction on the test dataset. The results disclose that the proposed ensemble model is sufficiently accurate in differentiating between primary emotions (such as sadness, fear, and anger) and secondary emotions. The suggested ensemble model achieved accuracy of 76.2%, 99.4%, and 99.6% on the FER-2013, CK+, and FERG-DB datasets, respectively.

DYNAMIC NONLINEAR EXPRESSION RECOGNITION TECHNOLOGY USING NEURAL NETWORK AND ATTENTION MECHANISM

The purpose is to apply deep learning to facial expression recognition and improve the efficiency of facial expression recognition in real scenes: first, convolutional neural network (CNN) theoretical model is proposed. On this basis, a new CNN model is proposed, which has six learning layers, including three convolutional layers and three fully connected layers. Then, based on the shortcomings of the model, weight decay and dropout optimization methods are proposed. Moreover, the attention module is designed based on the theory of attention mechanism. After each convolutional layer, a hybrid attention module (channel domain attention and spatial domain attention) is added. Finally, the frame error rate (FER) 2013 dataset and Cohn–Kanade (CK)+ dataset are used to test and compare the performance of the model. The results show that after 50Epoch, the accuracy rate of the original model fluctuates greatly in the process of convergence; however, after attention mechanism is added, the fluctuation of the accuracy of the model is obviously reduced. In the CK+ dataset, the accuracy of each model is maintained at about 95%, while the accuracy of FER2013 dataset is about 71%. After attention mechanism is added, the recognition rate of dynamic nonlinear expression is higher than that of basic model. The recognition rate of the original model and the optimized model with attention mechanism for the dynamic nonlinear expression with occlusion decreases in varying degrees, but the reduction of the recognition rate of the latter is alleviated.

Research on image sentiment analysis technology based on sparse representation

Many methods based on deep learning have achieved amazing results in image sentiment analysis. However, these existing methods usually pursue high accuracy, ignoring the effect on model training efficiency. Considering that when faced with large-scale sentiment analysis tasks, the high accuracy rate often requires long experimental time. In view of the weakness, a method that can greatly improve experimental efficiency with only small fluctuations in model accuracy is proposed, and singular value decomposition (SVD) is used to find the sparse feature of the image, which are sparse vectors with strong discriminativeness and effectively reduce redundant information; The authors propose the Fast Dictionary Learning algorithm (FDL), which can combine neural network with sparse representation. This method is based on K-Singular Value Decomposition, and through iteration, it can effectively reduce the calculation time and greatly improve the training efficiency in the case of small fluctuation of accuracy. Moreover, the effectiveness of the proposed method is evaluated on the FER2013 dataset. By adding singular value decomposition, the accuracy of the test suite increased by 0.53%, and the total experiment time was shortened by 8.2%; Fast Dictionary Learning shortened the total experiment time by 36.3%.

Facial Expression Recognition Using a Temporal Ensemble of Multi-Level Convolutional Neural Networks

Emotion recognition is indispensable in human-machine interaction systems. It comprises locating facial regions of interest in images and classifying them into one of seven classes: angry, disgust, fear, happy, neutral, sad, and surprise. Despite several breakthroughs in image classification, particularly in facial expression recognition, this research area is still challenging, as sampling in the wild is a demanding task. In this study, a two-stage method is proposed for recognizing facial expressions given a sequence of images. At the first stage, all face regions are extracted in each frame, and essential information that would be helpful and related to human emotion is obtained. Then, the extracted features from the previous step are considered temporal data and are assigned to one of the seven basic emotions. In addition, a study of multi-level features is conducted in a convolutional neural network for facial expression recognition. Moreover, various network connections are introduced to improve the classification task. By combining the proposed network connections, superior results are obtained compared to state-of-the-art methods on the FER2013 dataset. Furthermore, the performance of our temporal model is better than that of the single architecture of the 2017 EmotiW challenge winner on the AFEW 7.0 dataset.

Facial expression recognition using adapted residual based deep neural network

Emotion on our face can determine our feelings, mental state and can directly impact our decisions. Humans are subjected to undergo an emotional change in relation to their living environment and or at a present circumstance. These emotions can be anger, disgust, fear, sadness, happiness, surprise or neutral. Due to the intricacy and nuance of facial expressions and their relationship to emotions, accurate facial expression identification remains a difficult undertaking. As a result, we provide an end-to-end system that uses residual blocks to identify emotions and improve accuracy in this research field. After receiving a facial image, the framework returns its emotional state. The accuracy obtained on the test set of FERGIT dataset (an extension of the FER2013 dataset with 49300 images) was 75%. This proves the efficiency of the model in classifying facial emotions as this database poses a bunch of challenges such as imbalanced data, intraclass variance, and occlusion. To ensure the performance of our model, we also tested it on the CK+ database and its output accuracy was 97% on the test set.

RASN: Using Attention and Sharing Affinity Features to Address Sample Imbalance in Facial Expression Recognition

The sample imbalance of expression datasets always leads to poor recognition results for minority classes. To solve this problem, we propose a facial expression recognition network, called Residual Attentive Sharing Network (RASN). There is a fact that different expressions have affinity features, which makes it possible for the minority classes to benefit from the majority classes in the expression feature extraction process, from which we propose a sharing affinity features module to compensate for the inadequate feature learning of minority classes by sharing affinity features. In addition, an affinity features attention module is added to highlight expression-related affinity features and suppress expression-unrelated ones for enhancing the role of sharing affinity features. Experiments on the CK+, RAF-DB, and FER2013 datasets validate the robustness of our method to sample imbalance. The validation accuracies of our method are 96.97% on CK+, 71.44% on FER2013, and 90.91% on RAF-DB, respectively, which exceed current state-of-the-art methods.

Hybrid Deep Neural Network for Facial Expressions Recognition

Facial expressions are critical indicators of human emotions where recognizing facial expressions has captured the attention of many academics, and recognition of expressions in natural situations remains a challenge due to differences in head position, occlusion, and illumination. Several studies have focused on recognizing emotions from frontal images only, while in this paper wild images from the FER2013 dataset have been used to make a more generalizing model with the existence of its challenges, it is among the most difficult datasets that only got 65.5 % accuracy human-level. This paper proposed a model for recognizing facial expressions using pre-trained deep convolutional neural networks and the technique of transfer learning. this hybrid model used a combination of two pre-trained deep convolutional neural networks, training the model in multiple cases for more efficiency to categorize the facial expressions into seven classes. The results show that the best accuracy of the suggested models is 74.39% for the hybrid model, and 73.33% for Fine-tuned the single EfficientNetB0 model, while the highest accuracy for previous methods was 73.28%. Thus, the hybrid and single models outperform other state of art classification methods without using any additional, the hybrid and single models ranked in the first and second position among these methods. Also, The hybrid model has even outperformed the second-highest in accuracy method which used extra data. The incorrectly labeled images in the dataset unfairly reduce accuracy but our best model recognized their actual classes correctly.

Improved Facial Expression Recognition Method Based on GAN

Recognizing facial expressions accurately and effectively is of great significance to medical and other fields. Aiming at problem of low accuracy of face recognition in traditional methods, an improved facial expression recognition method is proposed. The proposed method conducts continuous confrontation training between the discriminator structure and the generator structure of the generative adversarial networks (GANs) to ensure enhanced extraction of image features of detected data set. Then, the high-accuracy recognition of facial expressions is realized. To reduce the amount of calculation, GAN generator is improved based on idea of residual network. The image is first reduced in dimension and then processed to ensure the high accuracy of the recognition method and improve real-time performance. Experimental part of the thesis uses JAFEE dataset, CK + dataset, and FER2013 dataset for simulation verification. The proposed recognition method shows obvious advantages in data sets of different sizes. The average recognition accuracy rates are 96.6%, 95.6%, and 72.8%, respectively. It proves that the method proposed has a generalization ability.

Enhancement of Patient Facial Recognition through Deep Learning Algorithm: ConvNet.

The use of machine learning algorithms for facial expression recognition and patient monitoring is a growing area of research interest. In this study, we present a technique for facial expression recognition based on deep learning algorithm: convolutional neural network (ConvNet). Data were collected from the FER2013 dataset that contains samples of seven universal facial expressions for training. The results show that the presented technique improves facial expression recognition accuracy without encoding several layers of CNN that lead to a computationally costly model. This study proffers solutions to the issues of high computational cost due to the implementation of facial expression recognition by providing a model close to the accuracy of the state-of-the-art model. The study concludes that deep l\\earning-enabled facial expression recognition techniques enhance accuracy, better facial recognition, and interpretation of facial expressions and features that promote efficiency and prediction in the health sector.

Efficient CNN Approach for Facial Expression Recognition

In the last decade, the Facial Expression Recognition field has been studied widely and become the base for many researchers, and still challenging in computer vision. Machine learning technique used in facial expression recognition facing many problems, since human emotions expressed differently from one to another. Nevertheless, Deep learning that represents a novel area of research within machine learning technology has the ability for classifying people’s faces into different emotion classes by using a Deep Neural Network (DNN). The Convolution Neural Network (CNN) method has been used widely and proved as very efficient in the facial expression recognition field. In this study, a CNN technique for facial expression recognition has been presented. The performance of this study has been evaluated using the fer2013 dataset, the total number of images has been used. The accuracy of each epoch has been tested which is trained on 29068 samples, validate on 3589 samples. The overall accuracy of 69.85% has been obtained for the proposed method.

Convolutional Neural Network With Batch Normalization for Classification of Emotional Expressions Based on Facial Images

Emotion recognition through facial images is one of the most challenging topics in human psychological interactions with machines. Along with advances in robotics, computer graphics, and computer vision, research on facial expression recognition is an important part of intelligent systems technology for interactive human interfaces where each person may have different emotional expressions, making it difficult to classify facial expressions and requires training data. large, so the deep learning approach is an alternative solution., The purpose of this study is to propose a different Convolutional Neural Network (CNN) model architecture with batch normalization consisting of three layers of multiple convolution layers with a simpler architectural model for the recognition of emotional expressions based on human facial images in the FER2013 dataset from Kaggle. The experimental results show that the training accuracy level reaches 98%, but there is still overfitting where the validation accuracy level is still 62%. The proposed model has better performance than the model without using batch normalization.

MIFAD-Net: Multi-Layer Interactive Feature Fusion Network With Angular Distance Loss for Face Emotion Recognition.

Understanding human emotions and psychology is a critical step toward realizing artificial intelligence, and correct recognition of facial expressions is essential for judging emotions. However, the differences caused by changes in facial expression are very subtle, and different expression features are less distinguishable, making it difficult for computers to recognize human facial emotions accurately. Therefore, this paper proposes a novel multi-layer interactive feature fusion network model with angular distance loss. To begin, a multi-layer and multi-scale module is designed to extract global and local features of facial emotions in order to capture part of the feature relationships between different scales, thereby improving the model's ability to discriminate subtle features of facial emotions. Second, a hierarchical interactive feature fusion module is designed to address the issue of loss of useful feature information caused by layer-by-layer convolution and pooling of convolutional neural networks. In addition, the attention mechanism is also used between convolutional layers at different levels. Improve the neural network's discriminative ability by increasing the saliency of information about different features on the layers and suppressing irrelevant information. Finally, we use the angular distance loss function to improve the proposed model's inter-class feature separation and intra-class feature clustering capabilities, addressing the issues of large intra-class differences and high inter-class similarity in facial emotion recognition. We conducted comparison and ablation experiments on the FER2013 dataset. The results illustrate that the performance of the proposed MIFAD-Net is 1.02–4.53% better than the compared methods, and it has strong competitiveness.

Performance Comparison of Different Convolutional Neural Network Approaches for Facial Expression Recognition

Abstract: Facial expression is a non-verbal way of communication to express the human state of mind using facial muscles. Happiness, sadness, anger, surprise, disgust, fear, and neutral expressions are widely used in the field of medical rehabilitation, sentiment analysis, counseling, and so on inspiring researchers to develop effective models to classify the expressions effectively. LeNet5, AlexNet, Deep Model, Shallow Model, Deep CNN Model are some commonly used models that have been developed to recognize facial expressions using machine learning and deep learning. In this research, a new convolutional neural network model has been proposed and compared with the existing models. The FER-2013 dataset has been used to evaluate the performance using different metrics to find the efficiency of the models. The proposed model provides comparatively better accuracy than most of the existing models, which is 64.4%. Keywords: Facial Expression, Image Classification, Convolutional Neural Network, Deep Learning, Non-verbal communication.

Facial Expression Emotion Recognition Model Integrating Philosophy and Machine Learning Theory.

Facial expression emotion recognition is an intuitive reflection of a person’s mental state, which contains rich emotional information, and is one of the most important forms of interpersonal communication. It can be used in various fields, including psychology. As a celebrity in ancient China, Zeng Guofan’s wisdom involves facial emotion recognition techniques. His book Bing Jian summarizes eight methods on how to identify people, especially how to choose the right one, which means “look at the eyes and nose for evil and righteousness, the lips for truth and falsehood; the temperament for success and fame, the spirit for wealth and fortune; the fingers and claws for ideas, the hamstrings for setback; if you want to know his consecution, you can focus on what he has said.” It is said that a person’s personality, mind, goodness, and badness can be showed by his face. However, due to the complexity and variability of human facial expression emotion features, traditional facial expression emotion recognition technology has the disadvantages of insufficient feature extraction and susceptibility to external environmental influences. Therefore, this article proposes a novel feature fusion dual-channel expression recognition algorithm based on machine learning theory and philosophical thinking. Specifically, the feature extracted using convolutional neural network (CNN) ignores the problem of subtle changes in facial expressions. The first path of the proposed algorithm takes the Gabor feature of the ROI area as input. In order to make full use of the detailed features of the active facial expression emotion area, first segment the active facial expression emotion area from the original face image, and use the Gabor transform to extract the emotion features of the area. Focus on the detailed description of the local area. The second path proposes an efficient channel attention network based on depth separable convolution to improve linear bottleneck structure, reduce network complexity, and prevent overfitting by designing an efficient attention module that combines the depth of the feature map with spatial information. It focuses more on extracting important features, improves emotion recognition accuracy, and outperforms the competition on the FER2013 dataset.

Facial Expression Recognition Method Combined with Attention Mechanism

Aiming at the slow speed and low accuracy of traditional facial expression recognition, a new method combining the attention mechanism is proposed. Firstly, group convolution is used to reduce network parameters. The channels of traditional convolution are grouped to cut off redundant connections so that the number of parameters decreases significantly. Secondly, the ERFNet network model was improved by combining the asymmetric residual module and the weak bottleneck module to improve the running speed and reduce the loss of accuracy. Finally, the attention mechanism was added into the feature extraction network to improve the recognition precision. The experiment shows that compared with traditional face recognition methods, the proposed method can improve the recognition precision and recall significantly; in CK+, Jaffe, and Fer2013 datasets, the recognition precision can reach 88.81%, 82.16%, and 79.33%, respectively.

Facial Expression Recognition Based on Transfer Learning and SVM

The facial expression datasets always have a problem: data with small amount or large amounts of data but also with large noisy. Both problems will affect the facial expression recognition accuracy of the model. A transfer learning method for facial expression recognition is proposed by combining the Convolutional Neural Network (CNN) and Support Vector Machine (SVM). SVM have good performance on small data sets and CNN based on transfer learning have better ability of feature extraction for large noisy data set. This method reduces the training time of model and increase the facial expression recognition accuracy. The experimental results show that the accuracy of the proposed method on the CK+ and FER2013 data sets has reached 99.6% and 68.1%.

Emotion Recognition using Convolutional Neural Network in Virtual Meeting Environment

This study is done in order to propose an Emotion Recognition System that uses Convolutional Neural Network in a Virtual Meeting Environment to detect non-verbal feedback that emerge when communicating. This study starts with the training process of the CNN model with version 2.3.0 of tensorflow-gpu library, along with FER-2013 dataset, where only 80% of the data is used as the training set, and the other 20% is used as the test set. The model is trained for 430 epochs that results in 73.86% rate of accuracy with a loss of 1.42. In the classification process, a Haar-Cascade Classifier algorithm is used to detect faces within an image that has been inputted using OpenCV. Next the already developed model is used to predict the image that has been pre-processed. Based on the results shown, it can be concluded that the study has provided satisfactory results and is expected to help in understanding non-verbal input given when communicating and among other various things.