Facial Emotion Classification Research Articles

Exploration of Facial Emotion Detection Systems Utilizing Convolutional Neural Networks: A Comprehensive Review

Facial emotion detection systems have witnessed significant advancements, particularly with the utilization of convolutional neural networks (CNNs). This paper provides a thorough survey of such systems, beginning with an introduction to artificial intelligence and the evolutionary trajectory of neural networks, including artificial neural networks (ANNs), recurrent neural networks (RNNs), and CNNs. The paper elaborates on CNNs' architecture and functionality, elucidating key components such as convolutional layers, pooling layers, and fully connected layers, while also spotlighting prominent CNN architectures like AlexNet and ResNet. It delineates the broad scope and diverse applications of facial emotion detection systems across various domains, including marketing research, crowd testing, AI robots, banking, and entertainment. In the literature review section, recent research papers on CNN models for facial expression recognition are synthesized, highlighting variances in datasets, methodologies, and accuracy levels. The paper concludes that CNNs represent the current pinnacle of facial emotion classification techniques, surpassing previous methodologies such as eigenfaces. It underscores the efficacy of deep CNN architectures trained on extensive facial image datasets in proficiently identifying emotions from facial expressions. Moreover, the paper emphasizes the necessity for ongoing endeavors to enhance accuracy, particularly concerning complex emotions like disgust. In essence, CNNs exhibit substantial promise for the development of real-world facial emotion detection systems, heralding a new era of sophisticated emotion recognition technology.

Robust CNN for facial emotion recognition and real-time GUI

<abstract> <p>Computer vision is witnessing a surge of interest in machines accurately recognizing and interpreting human emotions through facial expression analysis. However, variations in image properties such as brightness, contrast, and resolution make it harder for models to predict the underlying emotion accurately. Utilizing a robust architecture of a convolutional neural network (CNN), we designed an efficacious framework for facial emotion recognition that predicts emotions and assigns corresponding probabilities to each fundamental human emotion. Each image is processed with various pre-processing steps before inputting it to the CNN to enhance the visibility and clarity of facial features, enabling the CNN to learn more effectively from the data. As CNNs entail a large amount of data for training, we used a data augmentation technique that helps to enhance the model's generalization capabilities, enabling it to effectively handle previously unseen data. To train the model, we joined the datasets, namely JAFFE and KDEF. We allocated 90% of the data for training, reserving the remaining 10% for testing purposes. The results of the CCN framework demonstrated a peak accuracy of 78.1%, which was achieved with the joint dataset. This accuracy indicated the model's capability to recognize facial emotions with a promising level of performance. Additionally, we developed an application with a graphical user interface for real-time facial emotion classification. This application allows users to classify emotions from still images and live video feeds, making it practical and user-friendly. The real-time application further demonstrates the system's practicality and potential for various real-world applications involving facial emotion analysis.</p> </abstract>

NA

NA

Facial emotion classification based on the modified VGG and MobileNet

Emotion classification is an essential area of face recognition, which is widely applied recently, especially in information security. This study focus on model performance to explore a more efficient one that can produce a more precise prediction. In this paper, this study implement two distinct models, which are modified to classify emotions. The adaption models are added by one convolutional layer, one flatten layer, followed by activation layer and loss function layer. In order to estimate and compare them more straightforwardly, the metrics such as precision, loss, accuracy, recall and auc are introduced. Also, data augmentation to enlarge data sample size is achieved by rotation, width and length shift as well as horizontal lift. In the end, VGG16 shows a better performance with 0.9238 accuracy and 0.7780 loss in training, whereas MobileNet is slightly less outstanding with 0.9009 accuracy and 1.0280 loss in training. This study deal with choices of specific emotion classification models and attain a conclusion that VGG16 performs better in the given dataset.

Performance Enhancement in Facial Emotion Classification Through Noise-Injected FERCNN Model: A Comparative Analysis

Performance Enhancement in Facial Emotion Classification Through Noise-Injected FERCNN Model: A Comparative Analysis

A critical examination of deficiencies in the convolutional neural network model for facial emotion recognition

The challenge of addressing the issue of low accuracy in specific scenarios encountered during the implementation of facial emotion recognition systems arises due to the wide array of environments and varying conditions. In this study, the Facial Expression Recognition-2013 (FER-2013) dataset sourced from the Kaggle serves as the basis for training the models, with subsequent analysis conducted on the experimental outcomes. The dataset comprises a training set and a testing set, each annotated with labels representing seven distinct emotions, ranging from "angry" to "surprise". The models developed for facial emotion classification, tasked with automatically recognizing emotions based on provided images, consist of a MobileNet-based model and a self-built model employing convolutional neural networks. Both models exhibit an accuracy of approximately 60%, yet demonstrate deficiencies in predicting the "neutral" label. Additionally, the utilization of techniques such as confusion matrix and saliency map enable the comparative evaluation of model performance across different emotion labels and facilitates an analysis of their corresponding dominant facial regions. Based on a comparison of results obtained from representative cases, two potential factors contributing to these limitations are identified: a paucity of training data and the presence of ambiguous features. The findings of this study are expected to inform future directions for improvement and modification of facial emotion recognition models in order to enhance their applicability in diverse scenarios.

Facial Emotion Recognition and Classification Using the Convolutional Neural Network-10 (CNN-10)

The importance of facial expressions in nonverbal communication is significant because they help better represent the inner emotions of individuals. Emotions can depict the state of health and internal wellbeing of individuals. Facial expression detection has been a hot research topic in the last couple of years. The motivation for applying the convolutional neural network-10 (CNN-10) model for facial expression recognition stems from its ability to detect spatial features, manage translation invariance, understand expressive feature representations, gather global context, and achieve scalability, adaptability, and interoperability with transfer learning methods. This model offers a powerful instrument for reliably detecting and comprehending facial expressions, supporting usage in recognition of emotions, interaction between humans and computers, cognitive computing, and other areas. Earlier studies have developed different deep learning architectures to offer solutions to the challenge of facial expression recognition. Many of these studies have good performance on datasets of images taken under controlled conditions, but they fall short on more difficult datasets with more image diversity and incomplete faces. This paper applied CNN-10 and ViT models for facial emotion classification. The performance of the proposed models was compared with that of VGG19 and INCEPTIONV3. The CNN-10 outperformed the other models on the CK + dataset with a 99.9% accuracy score, FER-2013 with an accuracy of 84.3%, and JAFFE with an accuracy of 95.4%.

Data augmentation by using gans and image transformation in facial emotion classification

We propose different approach combinations including GANs and image transformation in facial emotion classification especially for FER-2013 dataset. Our goal is not to reach the highest accuracy for this dataset, but to compare and verify the data augmentation effects for several main GAN methods such as Vanilla GAN, CGAN, DCGAN and BiGAN. We used these GAN models to generate fake images and put them to our train dataset. And the data together was through image transformation to train in the CNN model that we designed to perform facial emotion classification. Base on the result when we used some GANs (BiGAN/DCGAN) and image transformation, the accuracy of the model indeed has been improved. The best combination data augmentation approaches in this paper are using image transformation and DCGAN.

Facial Micro Emotion Detection and Classification Using Swarm Intelligence based Modified Convolutional Network

Emotions are what makes us humans. Recognizing human emotions from facial micro expression features help us learn the true emotional state of a person. This technique of classifying the micro expressions can be used in varied application domains like criminology, marketing, job analysis, online learning etc. The field of recognizing micro emotions deals with tracking, recognizing, estimating & sequencing and classifying the recognized expressions. Artificial intelligence plays a crucial function in modern era of technology; micro expression analysis forms an ideal candidate of Deep Learning to correctly recognize these micro expressions when on display. The aim is to build a system that takes in a video data from any source and to recognize the micro expressions exhibited at various points in time. The challenge to overcome is to capture the fast changing expressions and to extract and align these facial features in order to extract suitable frames that provide the information from which the micro expressions can be ascertained by introducing it to a swarm optimization approach called the Artificial Bee Colony Approach. Implemented a novel approach that captures the essence of the micro expressions by an optical flow vector technique, that supplies its input to the modified deep learning Convolutional Neural Network, that in turn, is trained to categorize micro expressions on display. The Convolutional Neural Network combined with the Swarm Optimizer was able to achieve an accuracy of around 99.45% in identifying & classifying the facial micro expressions.

Comparisons of facial emotion recognition in different social contexts among patients with schizophrenia, major depressive disorder and bipolar disorder.

Previous studies have found that patients with schizophrenia (SCZ), major depressive disorder (MDD), and bipolar disorder (BD) all have facial emotion recognition deficits, but the differences and similarities of these deficits in the three groups of patients under different social interaction situations are not clear. The present study aims to compare the ability of facial emotion recognition in three different conversation situations from a cross-diagnostic perspective. Thirty-three participants with SCZ, 35 participants with MDD, and 30 participants with BD were recruited, along with 31 healthy controls. A computer-based task was given to assess the ability of Facial Emotion Categorization (FEC) under three different conversational situations (praise, blame, and inquiry). In the "praise" situation, patients with SCZ, MDD and BD were all slower to recognize anger emotion than the healthy controls. In all three clinical groups, patients with SCZ recognized angry faces faster than those with MDD and BD on a continuum from happy faces to angry faces in the "inquiry" situation, while no significant difference was found in the latter two groups. In addition, no significant defect was found in the percentage and threshold of angry face recognition in all three patient groups. Our findings indicate that patients with SCZ, MDD, and BD share both common and distinct deficits in facial emotion recognition during social interactions, which may be beneficial for early screening and precise intervention for these mental disorders.

Exploring an Affective and Responsive Virtual Environment to Improve Remote Learning

Online classes are typically conducted by using video conferencing software such as Zoom, Microsoft Teams, and Google Meet. Research has identified drawbacks of online learning, such as “Zoom fatigue”, characterized by distractions and lack of engagement. This study presents the CUNY Affective and Responsive Virtual Environment (CARVE) Hub, a novel virtual reality hub that uses a facial emotion classification model to generate emojis for affective and informal responsive interaction in a 3D virtual classroom setting. A web-based machine learning model is employed for facial emotion classification, enabling students to communicate four basic emotions live through automated web camera capture in a virtual classroom without activating their cameras. The experiment is conducted in undergraduate classes on both Zoom and CARVE, and the results of a survey indicate that students have a positive perception of interactions in the proposed virtual classroom compared with Zoom. Correlations between automated emojis and interactions are also observed. This study discusses potential explanations for the improved interactions, including a decrease in pressure on students when they are not showing faces. In addition, video panels in traditional remote classrooms may be useful for communication but not for interaction. Students favor features in virtual reality, such as spatial audio and the ability to move around, with collaboration being identified as the most helpful feature.

Facial Emotion Recognition through Neural Networks

Humans often express themselves through facial expressions. Deep learning techniques are used as an efficient system application process in research on the advancement of artificial intelligence technology in human-computer interactions. As an illustration, let’s say someone tries to communicate by using facial expressions. Some people who see it occasionally cannot foresee the expression or emotion it may evoke. Psychology includes study and evaluation of inferences in interpreting a person’s or group of people’s emotions when interacting in order to recognize emotions or facial expressions. Indeed, a convolutional neural networks (CNN) model may be learned to assess images and recognize facial expressions. This study suggests developing a system that can classify and forecast facial emotions using feature extraction and real-time Convolution Neural Network (CNN) technology from the OpenCV library. We have chosen FER 2013 Dataset as the main dataset for our study. Face detection, extraction of facial features, and facial emotion categorization are the three key procedures that make up the research that was implemented.

EMOSENSE – Multi-Modal Emotion Recognition to Identify Emotions

An extended study has been done over the past years to better comprehend human emotions. The embracement of technology to recognize and react to human emotions has become a required component of society. We present a fully functional multi-modal emotion recognition system in this study that integrates data from text, voice, facial expressions, and body language. In this study, the automatic classification of anger, fear, joy, sadness, surprise, disgust, and neutral emotions from text, facial expressions, voice, and body movements have been studied on the TESS, MELD, FER2013, and EDNLP datasets. Random Forest Classifier has been used for the classification of emotions using body language, VGG16 pre-trained model for facial emotion classification, Logistic Resgression for text emotion classification, and CNN for voice emotion classification. The logistic regression model for text emotion prediction leverages natural language processing (NLP) techniques to extract emotions from textual data. The CNN-based voice model utilizes speech recognition and emotion recognition algorithms to analyze audio signals and detect emotional cues in the speaker's voice. The facial expression model employs a combination of CNN-based VGG16 pre-trained model and modified convolutional layers to detect emotions. Meanwhile, the Random Forest Clasifier model is used to capture and interpret non-verbal cues, such as gestures, posture, and overall body movements, to enrich the emotion detection process. The real strength of our proposed system lies in its ability to synergistically combine information from multiple modalities.

Facial emotion perception abilities are related to grey matter volume in the culmen of cerebellum anterior lobe in drug-naïve patients with first-episode schizophrenia.

Impaired capability for understanding and interpreting the expressions on other people's faces manifests itself as a core feature of schizophrenia, contributing to social dysfunction. With the purpose of better understanding of the neurobiological basis of facial emotion perception deficits in schizophrenia, we investigated facial emotion perception abilities and regional structural brain abnormalities in drug-naïve patients with first-episode schizophrenia, and then examined the correlation between them. Fifty-two drug-naive patients with first-episode schizophrenia and 29 group-matched healthy controls were examined for facial emotion perception abilities assessed with the Facial Emotion Categorization and performed magnetic resonance imaging. The Facial Emotion Categorization data were inserted into a logistic function model so as to calculate shift point and slope as outcome measurements. Voxel-based morphometry was applied to investigate regional grey matter volume (GMV) alterations. The relationship between facial emotion perception and GMV was explored in patients using voxel-wise correlation analysis within brain regions that showed a significant GMV alterations in patients compared with controls. The schizophrenic patients performed differently on Facial Emotion Categorization tasks from the controls and presented a higher shift point and a steeper slope. Relative to the controls, patients showed GMV reductions in the superior temporal gyrus, middle occipital gyrus, parahippocampa gyrus, posterior cingulate, the culmen of cerebellum anterior lobe, cerebellar tonsil, and the declive of cerebellum posterior lobe. Importantly, abnormal performance on Facial Emotion Categorization was found correlated with GMV alterations in the culmen of cerebellum anterior lobe in schizophrenia. This study suggests that reduced GMV in the culmen of cerebellum anterior lobe occurs in first-episode schizophrenia, constituting a potential neuropathological basis for the impaired facial emotion perception in schizophrenia.

Facial Emotion Classifier using Convolutional Neural Networks for Reaction Review

Applications of facial emotion classification is gaining popularity in the world. There are many ways to train a model to classify human facial expressions by use of existing technologies. The strategy to order and recognize feelings of an individual conveyed by his facial expression is done by contrasting it to a gathered set of labelled experiences of feelings. In this paper, we propose the making of an intelligent system that will recognize and classify facial emotions. A multi-layer Convolutional Neural Network model is proposed. Another method of training using pretrained ResNet50 Model is explored. A basic live video streaming application is developed to showcase the use case of our model which will be capable of monitoring and recording facial emotions in real time from a live video stream and subsequently summarize the overall reactions at the end of the stream.

Relationship Between Facial Areas With the Greatest Increase in Non-local Contrast and Gaze Fixations in Recognizing Emotional Expressions

The aim of our study was to analyze gaze fixations in recognizing facial emotional expressions in comparison with o the spatial distribution of the areas with the greatest increase in the total (nonlocal) luminance contrast. It is hypothesized that the most informative areas of the image that getting more of the observer’s attention are the areas with the greatest increase in nonlocal contrast. The study involved 100 university students aged 19-21 with normal vision. 490 full-face photo images were used as stimuli. The images displayed faces of 6 basic emotions (Ekman’s Big Six) as well as neutral (emotionless) expressions. Observer’s eye movements were recorded while they were the recognizing expressions of the shown faces. Then, using a developed software, the areas with the highest (max), lowest (min), and intermediate (med) increases in the total contrast in comparison with the surroundings were identified in the stimulus images at different spatial frequencies. Comparative analysis of the gaze maps with the maps of the areas with min, med, and max increases in the total contrast showed that the gaze fixations in facial emotion classification tasks significantly coincide with the areas characterized by the greatest increase in nonlocal contrast. Obtained results indicate that facial image areas with the greatest increase in the total contrast, which preattentively detected by second-order visual mechanisms, can be the prime targets of the attention.

Deep cross feature adaptive network for facial emotion classification

Deep cross feature adaptive network for facial emotion classification

Facial Emotions Are Accurately Encoded in the Neural Signal of Those With Autism Spectrum Disorder: A Deep Learning Approach

BackgroundIndividuals with autism spectrum disorder (ASD) exhibit frequent behavioral deficits in facial emotion recognition (FER). It remains unknown whether these deficits arise because facial emotion information is not encoded in their neural signal or because it is encodes but fails to translate to FER behavior (deployment). This distinction has functional implications, including constraining when differences in social information processing occur in ASD, and guiding interventions (i.e., developing prosthetic FER vs. reinforcing existing skills). MethodsWe utilized a discriminative and contemporary machine learning approach—deep convolutional neural networks—to classify facial emotions viewed by individuals with and without ASD (N = 88) from concurrently recorded electroencephalography signals. ResultsThe convolutional neural network classified facial emotions with high accuracy for both ASD and non-ASD groups, even though individuals with ASD performed more poorly on the concurrent FER task. In fact, convolutional neural network accuracy was greater in the ASD group and was not related to behavioral performance. This pattern of results replicated across three independent participant samples. Moreover, feature importance analyses suggested that a late temporal window of neural activity (1000–1500 ms) may be uniquely important in facial emotion classification for individuals with ASD. ConclusionsOur results reveal for the first time that facial emotion information is encoded in the neural signal of individuals with (and without) ASD. Thus, observed difficulties in behavioral FER associated with ASD likely arise from difficulties in decoding or deployment of facial emotion information within the neural signal. Interventions should focus on capitalizing on this intact encoding rather than promoting compensation or FER prostheses.

A Review on Facial Emotion Recognition and Classification Analysis with Deep Learning

A Review on Facial Emotion Recognition and Classification Analysis with Deep Learning

A Review of the Effects of Valenced Odors on Face Perception and Evaluation.

How do valenced odors affect the perception and evaluation of facial expressions? We reviewed 25 studies published from 1989 to 2020 on cross-modal behavioral effects of odors on the perception of faces. The results indicate that odors may influence facial evaluations and classifications in several ways. Faces are rated as more arousing during simultaneous odor exposure, and the rated valence of faces is affected in the direction of the odor valence. For facial classification tasks, in general, valenced odors, whether pleasant or unpleasant, decrease facial emotion classification speed. The evidence for valence congruency effects was inconsistent. Some studies found that exposure to a valenced odor facilitates the processing of a similarly valenced facial expression. The results for facial evaluation were mirrored in classical conditioning studies, as faces conditioned with valenced odors were rated in the direction of the odor valence. However, the evidence of odor effects was inconsistent when the task was to classify faces. Furthermore, using a z-curve analysis, we found clear evidence for publication bias. Our recommendations for future research include greater consideration of individual differences in sensation and cognition, individual differences (e.g., differences in odor sensitivity related to age, gender, or culture), establishing standardized experimental assessments and stimuli, larger study samples, and embracing open research practices.