FER2013 Dataset Research Articles

Survey on Multimodal Emotion Detection

As we know, many people suffer from unhappiness in today’s world, so there is a requirement for a system that recommends music based on human emotions such as anger, sadness, happiness, etc. In this paper, we propose a system named Multimodal Emotion Detection which helps people by suggesting music and movies based on their emotions. We implemented this system using the FER-2013 dataset, which contains 35,887 images of different emotions such as sadness, happiness, and anger. We used Python’s TensorFlow framework and Haar Cascade algorithms to recommend movies and music based on human facial emotions. The main concern in existing recommendation systems is manual sorting. To avoid this, we propose this model, which automatically plays music and movies without requiring much browsing time

Online classroom student engagement analysis based on facial expression recognition using enhanced YOLOv5 for mitigating cyberbullying

Abstract The rapid expansion of online education has heightened concerns about cyberbullying in virtual classrooms. This paper presents a comprehensive approach for detecting cyberbullying by analyzing students' engagement and emotional responses in online classrooms. Due to the influence of camera resolution and surrounding light in online classrooms, students' facial expressions are often blurry, and the changes in facial features may not be significant. Moreover, most current recognition algorithms utilize larger models, which may not be suitable for real-time detection in online environments. To address these challenges, this study introduces a student facial expression recognition (SFER) method based on an enhanced YOLOv5 (you only look once version 5) model, termed SFER-YOLOv5. Firstly, the improved Soft-NMS is employed to replace the original non-maximum suppression (NMS), effectively enhancing training efficiency. Then, the coordinate attention (CA) module is incorporated into the backbone network to improve detection accuracy, particularly in classroom settings with multiple students or when students are at a considerable distance from the camera. Next, the efficient intersection over union (EIoU) loss function is utilized. EIoU calculates width and height losses separately based on complete IoU (CIoU), replacing the aspect ratio. Finally, Focal Loss is introduced to address sample imbalance issues. The comparative results show that SFER-YOLOv5 achieves an mAP@0.5 of 78.4% on the FER-2013 dataset, 98.1% on the CK+ dataset, and 88.9% on our self-constructed dataset (SFEC). These results underscore the effectiveness of SFER-YOLOv5 in enhancing the accuracy of student facial expression recognition. The proposed method detects reduced engagement, offering a preventive strategy for mitigating cyberbullying in virtual learning environments.

Implementation of Convolutional Neural Networks (CNN) in An Emotion Detection System for Measuring Learning Concentration Levels

Technological advancements have had a significant impact on the education sector, including the application of Convolutional Neural Networks (CNN) for facial image analysis. This research aims to implement CNN to measure students' learning concentration levels. The FER2013 dataset, which includes seven emotion classifications and comprises 28,709 images for training data, is used as the database. The data is processed through rescaling and augmentation to prepare the CNN model. The model consists of several convolutional layers, pooling layers, and fully connected layers designed to extract crucial features from facial images. Evaluation results demonstrate a very high accuracy of 94.95% on training data, indicating that the model effectively recognizes complex patterns within the data. Although there is a higher loss value of 157% and a decreased accuracy of 62.75% on validation data, this suggests that the model possesses a strong foundational capability and can still be improved through further adjustments. With high accuracy in training and promising validation results, the model shows substantial potential for real-world application, where it can assist teachers in understanding students' emotional responses in real-time. The implementation of CNN aids educators in comprehending students' emotional responses and adapting their teaching methods more effectively, thereby creating a more conducive learning environment and enhancing students' academic and social development. These findings also open opportunities for further research to improve the performance and generalization of the model on unseen data, making this technology an increasingly reliable tool in education

Research on facial expression recognition algorithm based on improved MobileNetV3

Aiming at the problem that face images are easily interfered by occlusion factors in uncontrollable environments, and the complex structure of traditional convolutional neural networks leads to low expression recognition rates, slow network convergence speed, and long network training time, an improved lightweight convolutional neural network is proposed for facial expression recognition algorithm. First, the dilation convolution is introduced into the shortcut connection of the inverted residual structure in the MobileNetV3 network to expand the receptive field of the convolution kernel and reduce the loss of expression features. Then, the channel attention mechanism SENet in the network is replaced by the two-dimensional (channel and spatial) attention mechanism SimAM introduced without parameters to reduce the network parameters. Finally, in the normalization operation, the Batch Normalization of the backbone network is replaced with Group Normalization, which is stable at various batch sizes, to reduce errors caused by processing small batches of data. Experimental results on RaFD, FER2013, and FER2013Plus face expression data sets show that the network reduces the training times while maintaining network accuracy, improves network convergence speed, and has good convergence effects.

RS-Xception: A Lightweight Network for Facial Expression Recognition

Facial expression recognition (FER) utilizes artificial intelligence for the detection and analysis of human faces, with significant applications across various scenarios. Our objective is to deploy the facial emotion recognition network on mobile devices and extend its application to diverse areas, including classroom effect monitoring, human–computer interaction, specialized training for athletes (such as in figure skating and rhythmic gymnastics), and actor emotion training. Recent studies have employed advanced deep learning models to address this task, though these models often encounter challenges like subpar performance and an excessive number of parameters that do not align with the requirements of FER for embedded devices. To tackle this issue, we have devised a lightweight network structure named RS-Xception, which is straightforward yet highly effective. Drawing on the strengths of ResNet and SENet, this network integrates elements from the Xception architecture. Our models have been trained on FER2013 datasets and demonstrate superior efficiency compared to conventional network models. Furthermore, we have assessed the model’s performance on the CK+, FER2013, and Bigfer2013 datasets, achieving accuracy rates of 97.13%, 69.02%, and 72.06%, respectively. Evaluation on the complex RAF-DB dataset yielded an accuracy rate of 82.98%. The incorporation of transfer learning notably enhanced the model’s accuracy, with a performance of 75.38% on the Bigfer2013 dataset, underscoring its significance in our research. In conclusion, our proposed model proves to be a viable solution for precise sentiment detection and estimation. In the future, our lightweight model may be deployed on embedded devices for research purposes.

Convolutional Neural Network-Based Digital Diagnostic Tool for the Identification of Psychosomatic Illnesses

This paper appraises convolutional neural network (CNN) models’ capabilities in emotion detection from facial expressions, seeking to aid the diagnosis of psychosomatic illnesses, typically made in clinical setups. Using the FER-2013 dataset, two CNN models were designed to detect six emotions with 64% accuracy—although not evenly distributed; they demonstrated higher effectiveness in identifying “happy” and “surprise.” The assessment was performed through several performance metrics—accuracy, precision, recall, and F1-scores—besides further validation with an additional simulated clinical environment for practicality checks. Despite showing promising levels for future use, this investigation highlights the need for extensive validation studies in clinical settings. This research underscores AI’s potential value as an adjunct to traditional diagnostic approaches while focusing on wider scope (broader datasets) plus focus (multimodal integration) areas to be considered among recommendations in forthcoming studies. This study underscores the importance of CNN models in developing psychosomatic diagnostics and promoting future development based on ethics and patient care.

Swin-FER: Swin Transformer for Facial Expression Recognition

The ability of transformers to capture global context information is highly beneficial for recognizing subtle differences in facial expressions. However, compared to convolutional neural networks, transformers require the computation of dependencies between each element and all other elements, leading to high computational complexity. Additionally, the large number of model parameters need extensive data for training so as to avoid overfitting. In this paper, according to the characteristics of facial expression recognition tasks, we made targeted improvements to the Swin transformer network. The proposed Swin-Fer network adopts the fusion strategy from the middle layer to deeper layers and employs a method of data dimension conversion to make the network perceive more spatial dimension information. Furthermore, we also integrated a mean module, a split module, and a group convolution strategy to effectively control the number of parameters. On the Fer2013 dataset, an in-the-wild dataset, Swin-Fer achieved an accuracy of 71.11%. On the CK+ dataset, an in-the-lab dataset, the accuracy reached 100%.

HSAW: A Half-Face Self-Attention Weighted Approach for Facial Expression Recognition

Facial expression recognition plays an increasingly important role in daily life, and it is used in several areas of human–computer interaction, such as robotics, assisted driving, and intelligent tutoring systems. However, the current mainstream methods are based on the whole face, and do not consider the existence of expression asymmetry between the left and right half-face. Hence, the accuracy of facial expression recognition needs to be improved. In this paper, we propose a half-face self-attention weighted approach called HSAW. Using statistical analysis and computer vision techniques, we found that the left half-face contains richer expression features than the right half-face. Specifically, we employed a self-attention mechanism to assign different weights to the left and right halves of the face. These weights are combined with convolutional neural network features for improved facial expression recognition. Furthermore, to attack the presence of uncertain categories in the dataset, we introduce adaptive re-labeling module, which can improve the recognition accuracy. Extensive experiments conducted on the FER2013 and RAF datasets have verified the effectiveness of the proposed method, which utilizes fewer parameters.

Facial Emotion Recognition System Using CNN for Song Mapping

This project introduces a novel system that integrates Facial Emotion Recognition (FER) with music mapping to enhance human-computer interaction. Leveraging a custom Convolutional Neural Network (CNN) algorithm, developed using the FER2013 dataset, we classify emotions into four categories- happy, sad, neutral, and angry- employing Haar Cascade for precise face detection and grayscale conversion for optimal CNN input. Our custom CNN demonstrates superior performance, achieving a testing accuracy of 77.23%, notably surpassing established models like VGG16 and ResNet50, which achieved 55.87% and 62.76% respectively. This system swiftly identifies emotions and recommends songs from the "278k Emotion Labeled Spotify Songs" playlist, aiming to boost user satisfaction. Through nuanced comparisons with VGG16 and ResNet50, our approach underscores its inherent strengths, suggesting promising advancements in Facial Emotion Recognition (FER) and music recommendation systems. With a focus on precision in emotion detection, our system subtly elevates user experience, contributing meaningfully to ongoing research in the field.

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model

AbstractThis paper presents a groundbreaking online educational platform that utilizes facial expression recognition technology to track the progress of students within the classroom environment. Through periodic image capture and facial data extraction, the platform employs ResNet50, CBAM, and TCNs for enhanced facial expression recognition. Achieving accuracies of 91.86%, 91.71%, 95.85%, and 97.08% on the RAF-DB, FER2013, CK + , and KDEF expression datasets, respectively, the proposed model surpasses the initial ResNet50 model in accuracy and detection of students' learning states. Comparative evaluations against state-of-the-art models using the RAF-DB, FER2013, CK + , and KDEF datasets underscore the significance of the results for educational institutions. By enhancing emotion recognition accuracy, improving feature relevance, capturing temporal dynamics, enabling real-time monitoring, and ensuring robustness and adaptability in online classroom environments, this approach offers valuable insights for educators to enhance teaching strategies and student outcomes. The combined capabilities of ResNet50, CBAM, and TCNs contribute uniquely to capturing dynamic changes in facial expressions over time, thereby facilitating accurate interpretation of students' emotions and engagement levels for more effective monitoring of learning behaviors in real-time.

Simple Histogram Equalization Technique Improves Performance of VGG Models on Facial Emotion Recognition Datasets

Facial emotion recognition (FER) is crucial across psychology, neuroscience, computer vision, and machine learning due to the diversified and subjective nature of emotions, varying considerably across individuals, cultures, and contexts. This study explored FER through convolutional neural networks (CNNs) and Histogram Equalization techniques. It investigated the impact of histogram equalization, data augmentation, and various model optimization strategies on FER accuracy across different datasets like KDEF, CK+, and FER2013. Using pre-trained VGG architectures, such as VGG19 and VGG16, this study also examined the effectiveness of fine-tuning hyperparameters and implementing different learning rate schedulers. The evaluation encompassed diverse metrics including accuracy, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), Area Under the Precision–Recall Curve (AUC-PRC), and Weighted F1 score. Notably, the fine-tuned VGG architecture demonstrated a state-of-the-art performance compared to conventional transfer learning models and achieved 100%, 95.92%, and 69.65% on the CK+, KDEF, and FER2013 datasets, respectively.

Mist–fog-assisted real-time emotion recognition using deep transfer learning framework for smart city 4.0

Face emotion recognition (FER) has found several applications across numerous industries. One such application area lies in the field of human–computer interaction, where it can improve user experiences by allowing systems to adjust based on users’ emotional states by leveraging the capabilities of the Internet of Things (IoT) to facilitate more responsive interactions. The increasing demand for precise and immediate emotion analysis across various applications drives the need for FER utilizing deep transfer learning (DTL). In this study, a novel Mist-Fog framework for real-time FER is introduced, which employs a DTL algorithm called Mobile-Net for quick and easy image classification on mobile and embedded devices. It addresses the challenges of operating deep neural networks (DNN) under limits on power and processing resources. By utilizing depth-wise separable convolutions, the MobileNet algorithm reduces computational complexity while maintaining reasonable accuracy on the FER-2013 dataset. A Fog-based architecture, in which a mist layer comprising edge devices like Raspberry-Pi (RPi), an ARM-based single-board computer, was considered for a smart city scenario. Further, the M/M/1/K queuing system was considered, which follows the First-Come-First-Serve (FCFS) scheduling criteria and a Markov Modulated Poisson Process (MMPP) arrival rate to evaluate the system’s performance. The effectiveness of this novel approach towards industrial computer vision paradigms is assessed pertaining to different performance criteria.

Autoencoder-based Image Denoiser Suitable for Image of Numbers with High Potential for Hardware Implementation

The algorithms for processing images and videos are currently essential for many applications. Many of these applications are specified for processing and analyzing images of numbers, such as smart meter reading, automated document processing, and processing of vehicles and license plate images in traffic monitoring and analysis. Consequently, eliminating noise is frequently used as a pre-processing step to improve subsequent analysis and processing outcomes. In this context, this manuscript proposes using artificial intelligence-based methods to increase the efficiency of the image-denoising process. However, the computational demands of these algorithms necessitate careful consideration of the hardware on which they are implemented. Therefore, this paper proposes using the simple autoencoder approach and evaluates its efficiency compared to the conventional methods. This unsupervised model is trained to identify and remove impulse noise from digital images by replacing some pixels with others from the outer dataset that can clarify the whole image more. The model was trained using handwritten numbers, MNIST, and data set size in the first trial and the FER2013 dataset in the second. The model is superior in the case of the simple dataset. Two versions of autoencoders are considered, the first with three layers and the second with five. The Traditional denoising methods are investigated for comparison purposes. The four conventional filtering procedures, AMF, DBMF, ADBMF, and MDBUTMF, are implemented using the MATLAB simulation environment, and the results are reported and compared with the proposed methodology. The results show that the proposed artificial intelligence-based method significantly outperforms the traditional methods regarding processing efficiency and the resulting image quality. Moreover, the computational intensity for the proposed methodology is chosen as a metric for evaluating the algorithm compliance for the hardware implementation compared to the other Artificial Intelligence (AI)-based denoising algorithms. The suggested technique has minor processing and training time compared to the other AI-based methods with adequate quality in case the images of numbers usually do not contain many details, making it more convenient for hardware implementation.

Facial Expression Recognition Using Convolutional Neural Network

Understanding others' intentions through nonverbal cues like facial emotions is crucial in human communication. To design and train Deep Learning Models, this paper describes in detail how Convolutional Neural Network Models are developed using tf. Keras. The aim is to Sort facial photos into one of the seven face detection classifiers, our model is developed in such a manner that it learns hidden nonlinearity from the entered facial images, which is vital for discriminating the form of emotion someone is expressing. The model proposed on the Lenet-5 architecture by Yann LeCun uses the subsampling, feature map, and activation function (ReLu) in between the convolutional layer and fully connected layer for the output soft-max activation function will be used. The FER-2013 dataset, which consists of 35,887 structured 48x48 pixel grayscale images, was used to train the CNN models. The training dataset has 28,709 elements, while the testing dataset has 3,589 elements, and while validation has 3,589 elements. Train and test are the two folder names used to organize the FER dataset. separated even further into distinct files, each holding a different kind of FER dataset class. To mitigate the overfitting of the dropout, batch normalization and the model are employed. Since this is a multiclass classification problem, we are utilizing the Soft-max activation function and the Rectified linear unit for non-linear operation (ReLu). We are training a categorical cross- entropy and matrix for accuracy based on the parameters to assess the constructed CNN model's performance by examining the training epoch history and we have used optimizer Adam (Adaptive Moment Estimation) with the learning rate of 0.0001. We obtained the accuracy of the LeNet-5 model on training at 95.49% and testing at 49.47% [13].

Comparison of CNN and ResNet Neural Networks on the Performance of Facial Expression Recognition

Facial expression recognition is a crucial task in numerous applications, including human-computer interaction, mental health monitoring, and human behavior analysis. Previous studies have primarily focused on individual models or techniques for improving emotion classification accuracy. However, a comparative analysis of different neural network architectures' performance for facial expression recognition is lacking. The main objective of this study is to compare the performance of Convolutional Neural Network (CNN) and Residual Network (ResNet) on the Fer2013 dataset. The author aims to analyze their behavior during the initial training phases and identify the architectural advantages and challenges associated with each model. Both models are trained using the same experimental setup to ensure a fair comparison. The models are trained using the Fer2013 dataset. The author employs a standard protocol for data preprocessing and augmentation. Results show that CNN achieves an accuracy of around 0.5 in the initial stages of training, which is significantly higher than ResNet's accuracy of 0.25. However, as training progresses, ResNet may outperform CNN because it has a more complicated structure that can capture more complex patterns. CNN exhibits superior performance during the initial training stage of the Fer2013 dataset. This reason may lie behind the fact that CNN has a simpler structure which makes it more sensitive on the basic features of the data.

SENSES-ASD: a social-emotional nurturing and skill enhancement system for autism spectrum disorder.

This article introduces the Social-Emotional Nurturing and Skill Enhancement System (SENSES-ASD) as an innovative method for assisting individuals with autism spectrum disorder (ASD). Leveraging deep learning technologies, specifically convolutional neural networks (CNN), our approach promotes facial emotion recognition, enhancing social interactions and communication. The methodology involves the use of the Xception CNN model trained on the FER-2013 dataset. The designed system accepts a variety of media inputs, successfully classifying and predicting seven primary emotional states. Results show that our system achieved a peak accuracy rate of 71% on the training dataset and 66% on the validation dataset. The novelty of our work lies in the intricate combination of deep learning methods specifically tailored for high-functioning autistic adults and the development of a user interface that caters to their unique cognitive and sensory sensitivities. This offers a novel perspective on utilising technological advances for ASD intervention, especially in the domain of emotion recognition.

Face Emotion Recognition System for Depression Detection Using AI Techniques

Abstract: The human face is an essential aspect of an individual's body. It plays a crucial function in detecting and identifying emotions since the face is where a person exhibits all of their fundamental emotions. Through human emotions, we solve different types of problems. Like healthcare, security, business, and education. The purpose of this paper is to present the detection of depression in the mental health sector. Depression or stress is faced by most of the population all over the world for many reasons and at different stages of life. As the current human life is a busy life cycle, a person gets depressed or stressed in their daily life. Depression may be found in educational activities, competitive or challenging tasks, employment pressure, family consequences, different sorts of human connection management, health issues, old age, and other situations. Artificial intelligence and deep learning approaches are suggested in this study to assess depression. This research is useful for analyzing the mental health of every employer and psychologist when counselling their patients. Here, we propose a deep convolutional neural network (DCNN) model. This model can classify two types of human facial emotions. Which is based on positive and negative emotions. This model is trained and tested using the FER-2013 dataset. The data set used for experimentation is a FER (Facial Expression Recognition) dataset available in the KAGGLE repository. The implementation environment includes Keras, TensorFlow, and OpenCV Python packages. The result includes the emotion detection accuracy between the training and test phases. The average accuracy achieved was 77%.

Monitoring and improving student attention using deep learning and wireless sensor networks

Enhancing students’ attention during lectures is crucial concern for educators. Traditionally, teachers assess student focus subjectively, which can lead to bias. We propose using neural networks to detect and assess students’ attention by analyzing their physiological factors and facial expressions. This approach can identify students losing focus and help them refocus without teacher intervention. Our proposed method involves two steps. Initially, we utilize the EduViT model with Self-Supervised Learning (SSL) to determine students’ focus level. Subsequently, we implement a system that monitors student attention using a camera mounted in the classroom. Additionally, a device attached to each student's desk displays their concentration level in three different tiers via Light-Emitting Diode (LED), alerting them if they're drifting. The EduViT model is based on the MobileViT architecture, which includes Squeeze-and-Excitation (SE) blocks to enhance data connections between channels compared to the original. To optimize the model's weight calculation and prevent data redundancy, we utilize the Self-Supervised Learning Barlow-Twin (SSL-BT) method after feature extraction on the CelebA dataset. Our proposed model achieves a maximum accuracy of 66.51% and an F1-score of 54.60% in recognizing various facial expressions in the FER2013 dataset. Finally, we export the training outcomes of our model in the “tflife” format and integrate them into a practical student attention evaluation system. 3-state LED lights attached to each student’s desk notify them of their concentration level, helping them stay focused and improve their study discipline.

A Transfer Learning Approach for Facial Emotion Recognition Using a Deep Learning Model

The facial expression recognition (FER) system is the process of identifying the emotional state of a person. Emotion recognition from facial expressions is a rapidly growing area of research with numerous applications, including psychology, marketing, and human-computer interaction. This paper presents a novel approach to FER employing deep learning techniques, specifically leveraging the power of transfer learning methodology. To address the challenge of accurately recognizing different emotions, including angry, disgusted, afraid, happy, sad, surprised, and neutral, from facial expressions, a pre-trained model MobileNetV2 architecture has been fine-tuned for improving accuracy. It is a lightweight convolutional neural network architecture, specifically designed for efficient on-device inference. For evaluating the performance of the proposed FER model, the state-of-the-art FER-2013 dataset, along random images and video clips has been employed. Experimental results demonstrate that the proposed FER model attained a remarkable accuracy rate exceeding 99% when tested on diverse sets of random images and video clips. Moreover, the system achieved a notable accuracy of 61% when evaluated against the FER-2013 dataset. Overall, this approach presents a significant advancement in the field of real-time facial expression recognition using the MobileNetV2 architecture with FER2013 dataset, which may improve the quality of human-computer interactions.

Improved optimizer with deep learning model for emotion detection and classification.

Facial emotion recognition (FER) is largely utilized to analyze human emotion in order to address the needs of many real-time applications such as computer-human interfaces, emotion detection, forensics, biometrics, and human-robot collaboration. Nonetheless, existing methods are mostly unable to offer correct predictions with a minimum error rate. In this paper, an innovative facial emotion recognition framework, termed extended walrus-based deep learning with Botox feature selection network (EWDL-BFSN), was designed to accurately detect facial emotions. The main goals of the EWDL-BFSN are to identify facial emotions automatically and effectively by choosing the optimal features and adjusting the hyperparameters of the classifier. The gradient wavelet anisotropic filter (GWAF) can be used for image pre-processing in the EWDL-BFSN model. Additionally, SqueezeNet is used to extract significant features. The improved Botox optimization algorithm (IBoA) is then used to choose the best features. Lastly, FER and classification are accomplished through the use of an enhanced optimization-based kernel residual 50 (EK-ResNet50) network. Meanwhile, a nature-inspired metaheuristic, walrus optimization algorithm (WOA) is utilized to pick the hyperparameters of EK-ResNet50 network model. The EWDL-BFSN model was trained and tested with publicly available CK+ and FER-2013 datasets. The Python platform was applied for implementation, and various performance metrics such as accuracy, sensitivity, specificity, and F1-score were analyzed with state-of-the-art methods. The proposed EWDL-BFSN model acquired an overall accuracy of 99.37 and 99.25% for both CK+ and FER-2013 datasets and proved its superiority in predicting facial emotions over state-of-the-art methods.