FER2013 Dataset Research Articles

An integrated approach for mental health assessment using emotion analysis and scales

AbstractDepression is a prominent cause of mental illness, which could primarily increase early death. It is possible that this is the root of suicidal ideation, and it causes severe impairment in daily life. By detecting human face traits, artificial intelligence (AI) has cleared the road for predicting human emotions. This predictive technique will be used to conduct a preliminary assessment of depression. Prediction is accomplished using a mixture of four modules namely Facial Emotion Recognition (FER), Scales Questionnaire, Speech Emotion Recognition (SER), and Doctor Chat. FER2013 dataset is used for the FER module, while for speech‐based recognition, RAVDESS, TESS, SAVEE, and CREMA‐D are collectively used. To improve the accuracy of the FER, the people in the given image will be fed into a Face API created with TensorFlow JS, which will eventually be given to the proposed model that will recognize human faces in the image. For SER, a python library known as Librosa is used for extracting audio features and it will be fed to the proposed model. The scales module of the app has questionnaires that can be answered, and the result can be generated based on the scores obtained using established scales used in modern psychology such as the HAM‐D, YMRS etc., Though deep learning can predict emotions, the user may choose to speak with a real doctor about the issues to clear up any doubts. The application has a Doctor Chat module, which is essentially a chat bot for interacting with a doctor. Using this module, the users can talk, exchange files, and have their questions answered. The accuracy of FER is 91% whereas for SER, it is 82% on the test sets. The proposed approach produces the highest accuracy for the benchmark dataset. These four modules will work together to produce a homogenous depression report.

A Lightweight Two-Stream Model for Driver Emotion Recognition

In recent years, more and more attention has been paid to the traffic safety of drivers. Studies show that the motor vehicle speed will affect the incidence of traffic accidents and traffic accident casualty rate. In this paper, machine learning vision technology is used for real-time recognition of driver emotions, and a lightweight SIC-NET+RC-NET dual-stream recognition model is designed to cope with different scenarios of vehicle speed during driving. SIC-NET is a lightweight CNN network model, which greatly reduces the number of model parameters and computational power requirements, sacrifices part of accuracy, and greatly improves the real-time performance of model recognition. To improve the residual convolutional network model of inception blocks, RC-NET nested multiple micro-convolutional neural networks are used to identity mapping between different inception blocks to enhance the network training effect, expand network width and convolutional depth, and alleviate network degradation. This article stipulates that when the vehicle speed is greater than 60km/h, SIC-NET is used. When the vehicle speed is less than or equal to 60km/h, SIC-NET+RC-NRT is used. The FER2013 dataset was used for testing and validation in this paper. When SIC-NET is used, the model has better real-time performance and can provide a more timely warning when the vehicle speed is higher. When SIC-NET+RC-NET is used, the accuracy of the model in the FER2013 dataset is 69.65%, and the model can better cope with the scenario of a low vehicle speed. The experiment shows that the lightweight dual-flow model has good robustness and identification effect, which solves the problem of using the same model for different vehicle speeds and can effectively ensure the safety of the driver’s life and property.

Facial expression recognition based on improved depthwise separable convolutional network

A single network model can’t extract more complex and rich effective features. Meanwhile, the network structure is usually huge, and there are many parameters and consume more space resources, etc. Therefore, the combination of multiple network models to extract complementary features has attracted extensive attention. In order to solve the problems existing in the prior art that the network model can’t extract high spatial depth features, redundant network structure parameters, and weak generalization ability, this paper adopts two models of Xception module and inverted residual structure to build the neural network. Based on this, a face expression recognition method based on improved depthwise separable convolutional network is proposed in the paper. Firstly, Gaussian filtering is performed by Canny operator to remove noise, and combined with two original pixel feature maps to form a three-channel image. Secondly, the inverted residual structure of MobileNetV2 model is introduced into the network structure. Finally, the extracted features are classified by Softmax classifier, and the entire network model uses ReLU6 as the nonlinear activation function. The experimental results show that the recognition rate is 70.76% in Fer2013 dataset (facial expression recognition 2013) and 97.92% in CK+ dataset (extended Cohn Kanade). It can be seen that this method not only effectively mines the deeper and more abstract features of the image, but also prevents network over-fitting and improves the generalization ability.

Deep Learning based Framework for Emotion Recognition using Facial Expression

Human convey their message in different forms. Expressing their emotions and moods through their facial expression is one of them. In this work, to avoid the traditional process of feature extraction (Geometry based method, Template based method, and Appearance based method), CNN model is used as a feature extractor for emotion detection using facial expression. In this study we also used three pre-trained models VGG-16, ResNet-50, Inception-V3. This Experiment is done on Fer-2013 facial expression dataset and Cohn Extended (CK+) dataset. By using FER-2013 dataset the accuracy rates for CNN, ResNet-50, VGG-16 and Inception-V3 are 76.74%, 85.71%, 85.78%s, 97.93% respectively. Similarly, the experimental results using CK+ dataset showed the accuracy rates for CNN, ResNet- 50, VGG-16 and Inception-V3 are 84.18%, 92.91%, 91.07%, and 73.16% respectively. The experimental results showed exceptional results for Inception-V3 with 97.93% using FER-2013 dataset and ResNet-50 with 91.92% using CK+ dataset.

An Intelligent Mental Health Identification Method for College Students: A Mixed-Method Study.

Mental health assessments that combine patients' facial expressions and behaviors have been proven effective, but screening large-scale student populations for mental health problems is time-consuming and labor-intensive. This study aims to provide an efficient and accurate intelligent method for further psychological diagnosis and treatment, which combines artificial intelligence technologies to assist in evaluating the mental health problems of college students. We propose a mixed-method study of mental health assessment that combines psychological questionnaires with facial emotion analysis to comprehensively evaluate the mental health of students on a large scale. The Depression Anxiety and Stress Scale-21(DASS-21) is used for the psychological questionnaire. The facial emotion recognition model is implemented by transfer learning based on neural networks, and the model is pre-trained using FER2013 and CFEE datasets. Among them, the FER2013 dataset consists of 48 × 48-pixel face gray images, a total of 35,887 face images. The CFEE dataset contains 950,000 facial images with annotated action units (au). Using a random sampling strategy, we sent online questionnaires to 400 college students and received 374 responses, and the response rate was 93.5%. After pre-processing, 350 results were available, including 187 male and 153 female students. First, the facial emotion data of students were collected in an online questionnaire test. Then, a pre-trained model was used for emotion recognition. Finally, the online psychological questionnaire scores and the facial emotion recognition model scores were collated to give a comprehensive psychological evaluation score. The experimental results of the facial emotion recognition model proposed to show that its classification results are broadly consistent with the mental health survey results. This model can be used to improve efficiency. In particular, the accuracy of the facial emotion recognition model proposed in this paper is higher than that of the general mental health model, which only uses the traditional single questionnaire. Furthermore, the absolute errors of this study in the three symptoms of depression, anxiety, and stress are lower than other mental health survey results and are only 0.8%, 8.1%, 3.5%, and 1.8%, respectively. The mixed method combining intelligent methods and scales for mental health assessment has high recognition accuracy. Therefore, it can support efficient large-scale screening of students' psychological problems.

Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces.

Owing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pandemic. In this study, we propose a graph-based emotion recognition method that adopts landmarks on the upper part of the face. Based on the proposed approach, several pre-processing steps were applied. After pre-processing, facial expression features need to be extracted from facial key points. The main steps of emotion recognition on masked faces include face detection by using Haar-Cascade, landmark implementation through a media-pipe face mesh model, and model training on seven emotional classes. The FER-2013 dataset was used for model training. An emotion detection model was developed for non-masked faces. Thereafter, landmarks were applied to the upper part of the face. After the detection of faces and landmark locations were extracted, we captured coordinates of emotional class landmarks and exported to a comma-separated values (csv) file. After that, model weights were transferred to the emotional classes. Finally, a landmark-based emotion recognition model for the upper facial parts was tested both on images and in real time using a web camera application. The results showed that the proposed model achieved an overall accuracy of 91.2% for seven emotional classes in the case of an image application. Image based emotion detection of the proposed model accuracy showed relatively higher results than the real-time emotion detection.

An Expression Recognition Model Based on Channel and Spatial Attention Fusion

Expression recognition is an essential subfield in biometric recognition as well as a cutting-edge research area in computer vision. It has several uses in autonomous driving, human-computer interface, autism research, and other fields. This study offers an expression recognition model based on fusion attention that extracts expression features from the channel and spatial domains, respectively, then combines the two features. It is then trained on the FER2013 and ExpW datasets. Experiment findings suggest that the model may greatly enhance face emotion recognition accuracy.

Enhanced Sample Self-Revised Network for Cross-Dataset Facial Expression Recognition

Recently, cross-dataset facial expression recognition (FER) has obtained wide attention from researchers. Thanks to the emergence of large-scale facial expression datasets, cross-dataset FER has made great progress. Nevertheless, facial images in large-scale datasets with low quality, subjective annotation, severe occlusion, and rare subject identity can lead to the existence of outlier samples in facial expression datasets. These outlier samples are usually far from the clustering center of the dataset in the feature space, thus resulting in considerable differences in feature distribution, which severely restricts the performance of most cross-dataset facial expression recognition methods. To eliminate the influence of outlier samples on cross-dataset FER, we propose the enhanced sample self-revised network (ESSRN) with a novel outlier-handling mechanism, whose aim is first to seek these outlier samples and then suppress them in dealing with cross-dataset FER. To evaluate the proposed ESSRN, we conduct extensive cross-dataset experiments across RAF-DB, JAFFE, CK+, and FER2013 datasets. Experimental results demonstrate that the proposed outlier-handling mechanism can reduce the negative impact of outlier samples on cross-dataset FER effectively and our ESSRN outperforms classic deep unsupervised domain adaptation (UDA) methods and the recent state-of-the-art cross-dataset FER results.

Local Multi-Head Channel Self-Attention for Facial Expression Recognition

Since the Transformer architecture was introduced in 2017, there has been many attempts to bring the self-attention paradigm in the field of computer vision. In this paper, we propose LHC: Local multi-Head Channel self-attention, a novel self-attention module that can be easily integrated into virtually every convolutional neural network, and that is specifically designed for computer vision, with a specific focus on facial expression recognition. LHC is based on two main ideas: first, we think that in computer vision, the best way to leverage the self-attention paradigm is the channel-wise application instead of the more well explored spatial attention. Secondly, a local approach has the potential to better overcome the limitations of convolution than global attention, at least in those scenarios where images have a constant general structure, as in facial expression recognition. LHC-Net achieves a new state-of-the-art in the FER2013 dataset, with a significantly lower complexity and impact on the “host” architecture in terms of computational cost when compared with the previous state-of-the-art.

Facial expression recognition via a jointly-learned dual-branch network

Human emotion recognition depends on facial expressions, and essentially on the extraction of relevant features. Accurate feature extraction is generally difficult due to the influence of external interference factors and the mislabelling of some datasets, such as the Fer2013 dataset. Deep learning approaches permit an automatic and intelligent feature extraction based on the input database. But, in the case of poor database distribution or insufficient diversity of database samples, extracted features will be negatively affected. Furthermore, one of the main challenges for efficient facial feature extraction and accurate facial expression recognition is the facial expression datasets, which are usually considerably small compared to other image datasets. To solve these problems, this paper proposes a new approach based on a dual-branch convolutional neural network for facial expression recognition, which is formed by three modules: The two first ones ensure features engineering stage by two branches, and features fusion and classification are performed by the third one. In the first branch, an improved convolutional part of the VGG network is used to benefit from its known robustness, the transfer learning technique with the EfficientNet network is applied in the second branch, to improve the quality of limited training samples in datasets. Finally, and in order to improve the recognition performance, a classification decision will be made based on the fusion of both branches’ feature maps. Based on the experimental results obtained on the Fer2013 and CK+ datasets, the proposed approach shows its superiority compared to several state-of-the-art results as well as using one model at a time. Those results are very competitive, especially for the CK+ dataset, for which the proposed dual branch model reaches an accuracy of 99.32, while for the FER-2013 dataset, the VGG-inspired CNN obtains an accuracy of 67.70, which is considered an acceptable accuracy, given the difficulty of the images of this dataset.

Facial Emotion Recognition in Verbal Communication Based on Deep Learning.

Facial emotion recognition from facial images is considered a challenging task due to the unpredictable nature of human facial expressions. The current literature on emotion classification has achieved high performance over deep learning (DL)-based models. However, the issue of performance degradation occurs in these models due to the poor selection of layers in the convolutional neural network (CNN) model. To address this issue, we propose an efficient DL technique using a CNN model to classify emotions from facial images. The proposed algorithm is an improved network architecture of its kind developed to process aggregated expressions produced by the Viola–Jones (VJ) face detector. The internal architecture of the proposed model was finalised after performing a set of experiments to determine the optimal model. The results of this work were generated through subjective and objective performance. An analysis of the results presented herein establishes the reliability of each type of emotion, along with its intensity and classification. The proposed model is benchmarked against state-of-the-art techniques and evaluated on the FER-2013, CK+, and KDEF datasets. The utility of these findings lies in their application by law-enforcing bodies in smart cities.

INFORMATION MANAGEMENT SYSTEM FOR MONITORING THE EMOTIONAL STATE OF STUDENTS USING AI

In this current changing era, everything has become digitalised, education is conducted online. Teachers use online platforms to teach their coursework, but it has recently been realised that the classes were not conducted efficiently. The challenge here was to know if the students were understanding the concepts or not. In an offline class, the teacher could get the information just by looking at the students. However, in an online class, it is not possible to look into each student and get collective data. So, in order to cope with these changing demands of the era, we have attempted to build a system that can be helpful in analysing the emotions of students. A framework is built with the convolutional neural network (CNN). The network is trained on the FER2013 dataset and the emotions are analysed and live feedback is given to the teachers by showing the statistics. That represents the proportion of feelings that are angry, disgusted, afraid, joyful, neutral, sad, and surprised. In addition to that, the recording of the complete online class with the label of individual emotions is displayed in the recorded video, and the videos are compressed, so whenever the teachers wish to see the recorded emotions of the class. They need not view the 1-hour video for a 1-hour class, the video is compressed to a few minutes. This saves both time and space.

Facial Expression Recognition Based on Improved Residual Block Network

<p>Facial expression recognition is widely used, but there are some problems such as complex scenes, lack of data sets and low recognition rate. In this paper, we construct a new network model and name it RNFC. The RNFC network adopts 6 improved residual blocks to extract features. Features are passed into the fully connected layer by flattening the data, and Dropout techniques are introduced between the fully connected layers to prevent overfitting of the model. Based on the pytorch framework, we use a cross-entropy loss function to improve the training speed of the network. And perform denoising and enhancement pre-processing on the FER2013 dataset. The RNFC network is trained and tested on the pretreated FER2013. It has a higher recognition rate than classical networks such as VGGnet19 and ResNet18.</p> <p> </p>

An Approach for Learner Categorization Based on Emotions in Intelligent Adaptive E-Learning Environment

The pandemic across the globe has constrained the change from a conventional face to face to e-learning platforms. The most challenging task during online learning is to be aware and support the emotional side of students. In existing environments, the emotion of the listener consideration is lagging. This can be provided by capturing the emotions of the listener through facial expressions. In general, the most common facial expressions are happy, sad, anger, fear, disgust, neutral and surprise. This knowledge can be used to classify different listeners. Hence in this article, we proposed a novel approach to identify an emotion based learner category in the development of Intelligent Adaptive E-Learning Environment by using Convolution Neural Network. The major work is composed of emotion detection model and learner categorization. The emotion detection model is trained by using a standard FER2013 dataset and it is extended with live streams of learners. The results of emotion detection model are extended to categorize the learners by fusing emotions and comprehend as Active, Evaluative, Passive and Non-Listener. The proposed model is trained using 100 epochs and achieved an accuracy of 94.44% in the training phase. This knowledge helps to interpret learner’s participation in e-learning environment.

Target Detection and Network Optimization: Deep Learning in Face Expression Feature Recognition

Facial expression recognition technology has been more and more in demand in security, entertainment, education, medical, and other domains as artificial intelligence has advanced, and face expression recognition technology based on deep learning has become one of the research hotspots. However, there are still some issues with the existing deep learning convolutional neural network; the feature extraction technique has to be improved, and the design of the detailed network structure needs to be optimized. It is critical to do more research on the deep learning convolutional neural network model in order to increase the accuracy of face facial expression detection. In this paper, a deep learning convolutional neural network structure combining VGG16 convolutional neural network and long and short-term memory networks is designed to address the shortcomings of existing deep learning methods in face expression recognition, which are prone to overfitting and gradient disappearance, resulting in low test accuracy. This structure easily and effectively collects facial expression information and then classifies the retrieved features using a support vector machine to detect face expressions. Finally, the fer2013 dataset is used to train face expression recognition, and the results demonstrate that the built deep convolutional neural network model can effectively increase face expression identification accuracy.

Emotion detection from facial expression using image processing

Abstract: Facial expression recognition is a powerful tool for communicating our emotions, understanding, and intent with each other. It is an intelligent human-computer interaction technology. Various studies have been conducted to classify facial expressions. Six fundamental universal emotions can be expressed through facial expressions: happiness, sadness, anger, fearful, surprised, and neutral. In this project, emotion detection can be implemented in real time with the help of a webcam. Our work proposed a CNN-based VGG16 architecture for emotion detection systems. A model would be trained by using the FER-2013 dataset. Then the images from the dataset are first pre-processed, which includes operations such as image scaling, changing the colour mode, and so on. Following that, a CNN model with multiple layers was created. After that, the model would be trained with the specified dataset, resulting in the .h5 file, which is a pre-trained model file. Instead of repeatedly training the model, the results can be predicted using this file.

Texture based feature extraction using symbol patterns for facial expression recognition.

Facial expressions can convey the internal emotions of a person within a certain scenario and play a major role in the social interaction of human beings. In automatic Facial Expression Recognition (FER) systems, the method applied for feature extraction plays a major role in determining the performance of a system. In this regard, by drawing inspiration from the Swastik symbol, three texture based feature descriptors named Symbol Patterns (SP1, SP2 and SP3) have been proposed for facial feature extraction. SP1 generates one pattern value by comparing eight pixels within a 33 neighborhood, whereas, SP2 and SP3 generates two pattern values each by comparing twelve and sixteen pixels within a 55 neighborhood respectively. In this work, the proposed Symbol Patterns (SP) have been evaluated with natural, fibonacci, odd, prime, squares and binary weights for determining the optimal recognition accuracy. The proposed SP methods have been tested on MUG, TFEID, CK+, KDEF, FER2013 and FERG datasets and the results from the experimental analysis demonstrated an improvement in the recognition accuracy when compared to the existing FER methods.

Emotion Recognition of Online Education Learners by Convolutional Neural Networks.

At present, the facial expression recognition model in video communication has problems such as weak network generalization ability and complex model structure, which leads to a large amount of computation. Firstly, the Inception architecture is adopted as a design philosophy. The Visual Geometry Group Network (VGGNet) model is improved. Multiscale kernel convolutional layers are constructed to obtain more expressive features. Secondly, the attention mechanism is integrated into a multiscale feature fusion network to form a multiattention mechanism Convolutional Neural Network (CNN) model. Novel spatial and multichannel attention models are designed. The effects of redundant information and noise are reduced. Finally, experiments are carried out on the Fer2013 dataset and the Extended Cohn-Kanade Dataset (CK+) to verify the detection accuracy of the model. The results show that the Delivered Duty Unpaid (DDU) loss can be used for facial expression recognition in complex environments. After the attention module is added, the overall recognition accuracy of the network on Fer2013 and CK+ has been improved to varying degrees. The addition of the channel attention module has a more obvious effect on the recognition accuracy compared with the spatial attention module. The addition of the attention module enables the network to increase the attention to error-prone samples. The improved network model can better extract the key features of facial expressions, enhance the feature discrimination ability, and improve the recognition accuracy of error-prone expressions. The accuracy rate of facial expression recognition with larger movements is over 98%. Facial expressions are an important way of communication between people, and online video has greatly limited this communication method. The proposed CNN model based on multiscale feature fusion will effectively solve these network limitations and have an important and positive impact on future network information exchange.

Ensemble-based face expression recognition approach for image sentiment analysis

<span>Sentiment analysis based on images is an evolving area of study. Developing a reliable facial expression recognition (FER) device remains a difficult challenge as recognizing emotional feelings reflected in an image is dependent on a diverse set of factors. This paper presented an ensemble-based model for FER that incorporates multiple classification models: i) customized convolutional neural network (CNN), ii) ResNet50, and iii) InceptionV3. The model averaging ensemble classifier method is used to ensemble the predictions from the three models. Subsequently, the proposed FER model is trained and tested on a dataset with an uncontrolled environment (FER-2013 dataset). The experiment demonstrated that ensembling multiple classifiers outperformed all single classifiers in classifying positive and neutral expressions (91.7%, 81.7% and 76.5% accuracy rate for happy, surprise, and neutral, respectively). However, when classifying disgust, anger, and sadness, the ResNet50 model alone is the better choice. Although the Custom CNN performs the best in classifying fear expression (55.7% accuracy), the proposed FER model can still classify fear expression with comparable performance (52.8% accuracy). This paper demonstrated the potential of using the ensemble-based method to enhance the performance of FER. As a result, the proposed FER model has shown a 72.3% accuracy rate.</span>

Real‐time sentiment analysis of students based on mini‐Xception architecture for wisdom classroom

AbstractSentiment analysis has a wide application prospect in business, medicine, security and other fields, which provides a new perspective for the development of education. Students' sentiment data play an important role in the evaluation of teachers' teaching quality and students' learning effect, and provide a basis for the implementation of effective learning intervention. However, most of the research is to obtain the real‐time learning status of students in the classroom through teachers' naked eye observation and students' text feedback, which will lead to some problems such as incomplete feedback content and delayed feedback analysis. Based on the mini‐Xception framework, this article implements the real‐time identification and analysis of student sentiment in classroom teaching, and the degree of student engagement is analyzed according to the teaching events triggered by teacher to provide reasonable suggestions for subsequent teaching progress. The experimental results show that the mini‐Xception model trained by FER2013 data sets has high recognition accuracy for the real‐time detection of seven student sentiments, and the average accuracy is 76.71%. Compared with text feedback, it can assist teachers in understanding student learning states in time so that they can take corresponding actions, and realize the real‐time performance of wisdom classroom teaching information feedback, the high efficiency of information transmission, and the intelligence of information processing.