An optimal hybrid deep learning-aided facial emotion detection and classification scheme to identify criminal activities

Background/Objectives: There is only limited research work is going on in the field of facial expression recognition on low resolution images. Mostly, all the images in the real world will be in low resolution and might also contain noise, so this study is to design a novel convolutional neural network model (FERConvNet), which can perform better on low resolution images. Methods: We proposed a model and then compared with state-of-art models on FER2013 dataset. There is no publicly available dataset, which contains low resolution images for facial expression recognition (Anger, Sad, Disgust, Happy, Surprise, Neutral, Fear), so we created a Low Resolution Facial Expression (LRFE) dataset, which contains more than 6000 images of seven types of facial expressions. The existing FER2013 dataset and LRFE dataset were used. These datasets were divided in the ratio 80:20 for training and testing and validation purpose. A HDM is proposed, which is a combination of Gaussian Filter, Bilateral Filter and Non local means denoising Filter. This hybrid denoising method helps us to increase the performance of the convolutional neural network. The proposed model was then compared with VGG16 and VGG19 models. Findings: The experimental results show that the proposed FERConvNet_HDM approach is effective than VGG16 and VGG19 in facial expression recognition on both FER2013 and LRFE dataset. The proposed FERConvNet_HDM approach achieved 85% accuracy on Fer2013 dataset, outperforming the VGG16 and VGG19 models, whose accuracies are 60% and 53% on Fer2013 dataset respectively. The same FERConvNet_HDM approach when applied on LRFE dataset achieved 95% accuracy. After analyzing the results, our FERConvNet_HDM approach performs better than VGG16 and VGG19 on both Fer2013 and LRFE dataset. Novelty/Applications: HDM with convolutional neural networks, helps in increasing the performance of convolutional neural networks in Facial expression recognition. Keywords: Facial expression recognition; facial emotion; convolutional neural network; deep learning; computer vision

This study presents an enhanced convolutional neural network (CNN) architecture tailored for accurate facial expression recognition. The model is trained on the FER2013 dataset and evaluated using both FER2013 and a custom dataset containing natural facial expressions. By incorporating multiple convolutional and pooling layers along with dropout regularization, the network effectively extracts and classifies emotion-related features. Experimental results demonstrate high recognition accuracy and strong generalization across controlled and real-world image scenarios. In order to study the application of convolutional neural networks in facial expression recognition, a 10-layer convolutional neural network model is designed to recognize facial expressions. The last layer uses the Softmax function to output the classification results of expressions. First, the convolution and pooling algorithms of convolutional neural networks were studied and the structure of the model was designed. Secondly, in order to more vividly display the features extracted by the convolutional layer, the extracted features are visualized and displayed in the form of feature maps. The convolutional neural network model in this work was tested on the Fer-2013 data set, and the experimental results demonstrated the superiority of the recognition rate. It is known that the Fer-2013 dataset contains data collected in an experimental environment, and in order to verify the generalization ability of model recognition, a self-made facial expression data set in natural state was created, and performed a series of preprocessing on the face images such as cropping, grayscale and pixel adjustment. The trained model, which was previously applied to the Fer-2013 dataset, was tested out on the new dataset. The experiment yielded promising results, one of which in the form of a recognition accuracy rate as high as 85.1%.

Facial expression recognition is a challenging problem in image classification. Recently, the use of deep learning is gaining importance in image classification. This has led to increased efforts in solving the problem of facial expression recognition using convolutional neural networks (CNNs). A significant challenge in deep learning is to design a network architecture that is simple and effective. A simple architecture is fast to train and easy to implement. An effective architecture achieves good accuracy on the test data. CNN architectures are black boxes to us. VGGNet, AlexNet and Inception are well-known CNN architectures. These architectures have strongly influenced CNN model designs for new datasets. Almost all CNN models known to achieve high accuracy on facial expression recognition problem are influenced by these architectures. This work tries to overcome this limitation by using FER-2013 dataset as starting point to design new CNN models. In this work, the effect of CNN parameters namely kernel size and number of filters on the classification accuracy is investigated using FER-2013 dataset. Our major contribution is a thorough evaluation of different kernel sizes and number of filters to propose two novel CNN architectures which achieve a human-like accuracy of 65% (Goodfellow et al. in: Neural information processing, Springer, Berlin, pp 117–124, 2013) on FER-2013 dataset. These architectures can serve as a basis for standardization of the base model for the much inquired FER-2013 dataset.

Facial expression emotion recognition is an intuitive reflection of a person’s mental state, which contains rich emotional information, and is one of the most important forms of interpersonal communication. It can be used in various fields, including psychology. As a celebrity in ancient China, Zeng Guofan’s wisdom involves facial emotion recognition techniques. His book Bing Jian summarizes eight methods on how to identify people, especially how to choose the right one, which means “look at the eyes and nose for evil and righteousness, the lips for truth and falsehood; the temperament for success and fame, the spirit for wealth and fortune; the fingers and claws for ideas, the hamstrings for setback; if you want to know his consecution, you can focus on what he has said.” It is said that a person’s personality, mind, goodness, and badness can be showed by his face. However, due to the complexity and variability of human facial expression emotion features, traditional facial expression emotion recognition technology has the disadvantages of insufficient feature extraction and susceptibility to external environmental influences. Therefore, this article proposes a novel feature fusion dual-channel expression recognition algorithm based on machine learning theory and philosophical thinking. Specifically, the feature extracted using convolutional neural network (CNN) ignores the problem of subtle changes in facial expressions. The first path of the proposed algorithm takes the Gabor feature of the ROI area as input. In order to make full use of the detailed features of the active facial expression emotion area, first segment the active facial expression emotion area from the original face image, and use the Gabor transform to extract the emotion features of the area. Focus on the detailed description of the local area. The second path proposes an efficient channel attention network based on depth separable convolution to improve linear bottleneck structure, reduce network complexity, and prevent overfitting by designing an efficient attention module that combines the depth of the feature map with spatial information. It focuses more on extracting important features, improves emotion recognition accuracy, and outperforms the competition on the FER2013 dataset.

NA

Facial Emotion Recognition (FER) holds great importance in the fields of computer vision and machine learning. In this study, the aim is to improve the accuracy of facial expression recognition by incorporating attention mechanisms into Convolutional Neural Networks (CNN) with FER2013 dataset, which consists of grayscale images categorized into seven expressions. The combination of proposed CNN architecture and attention mechanisms is thoroughly elucidated, emphasizing the operations and interactions of their components. Additionally, the effectiveness of the new model is evaluated through experiments, comparing its performance with existing approaches in terms of accuracy. Besides, the results demonstrate that the CNN architecture with attention mechanisms outperforms the original CNN by achieving an improved accuracy rate of 69.07%, which is higher than 68.04% accuracy rate of original CNN. Moreover, the study further discusses the confusion matrix analysis, revealing the challenges faced in recognizing specific emotions due to limited training data and vague facial features. In the future, this study suggests addressing these limitations through data augmentation and to reduce the gap between training and testing accuracy. Overall, this research highlights the potential of attention mechanisms in enhancing facial expression recognition systems, paving the way for advanced applications in various domains.

Facial expression recognition (FER) has made much progress and is supported by many scientific studies conducted in the past decade. The technique and architecture model used in the FER are the aspects that get the most improvement from the researchers. One of the famously used techniques in conducting FER is Deep Convolutional Neural Network (DCNN). The development of DCNN architecture has a vital role in increasing the accuracy of facial expression recognition. The choice of architecture will also affect the total computational costs required to perform facial expression recognition activities. This paper compares some of the DCNN architectures in the past FER research during 2010-2020 using various FER dataset. This paper presents detailed information on several DCNN architectures in terms of the dataset preprocessing techniques and accuracy value for some accessible FER datasets such as Fer2013, CK+, AFEW, and other datasets. This research will explain other FER researchers who are new in FER research to determine the DCNN architecture used based on several FER datasets.

One of the ways humans communicate is by using facial expressions. Research on technology development in artificial intelligence uses deep learning methods in human and computer interactions as an effective system application process. One example, if someone does show and tries to recognize facial expressions when communicating. The prediction of the expression or emotion of some people who see it sometimes does not understand. In psychology, the detection of emotions or facial expressions requires analysis and assessment of decisions in predicting a person's emotions or group of people in communicating. This research proposes the design of a system that can predict and recognize the classification of facial emotions based on feature extraction using the Convolution Neural Network (CNN) algorithm in real-time with the OpenCV library, namely: TensorFlow and Keras. The research design implemented in the Raspberry Pi consists of three main processes, namely: face detection, facial feature extraction, and facial emotion classification. The prediction results of facial expressions in research with the Convolutional Neural Network (CNN) method using Facial Emotion Recognition (FER-2013) were 65.97% (sixty-five point ninety-seven percent).

Facial emotion recognition (FER) is the technology or process of identifying and interpreting human emotions based on the analysis of facial expressions. It involves using computer algorithms and machine learning techniques to detect and classify emotional states from images or videos of human faces. Further, FER plays a vital role in recognizing and understanding human emotions to better interpret someone’s feelings, intentions, and attitudes. In the present time, it is widely used in various fields such as healthcare, human–computer interaction, law enforcement, security, and beyond. FER has a wide range of practical applications across various industries including Emotion Monitoring, Adaptive Learning, and Virtual Assistants. This paper presents a comparative analysis of FER algorithms, focusing on deep learning approaches. The performance of different datasets, including FER2013, JAFFE, AffectNet, and Cohn–Kanade, is evaluated using convolutional neural networks (CNNs), deep face, attentional convolutional networks (ACNs), and deep belief networks (DBNs). Among the tested algorithms, DBNs outperformed other algorithms, reaching the highest accuracy of 98.82%. These results emphasize the effectiveness of deep learning techniques, particularly DBNs, in FER. Additionally, outlining the advantages and disadvantages of current research on facial emotion identification might direct future research efforts in the direction of the most profitable directions.

Facial emotion recognition (FER) stands as a pivotal artificial intelligence (AI)-driven technology that exploits the capabilities of computer-vision techniques for decoding and comprehending emotional expressions displayed on human faces. With the use of machine-learning (ML) models, specifically deep neural networks (DNN), FER empowers the automatic detection and classification of a broad spectrum of emotions, encompassing surprise, happiness, sadness, anger, and more. Challenges in FER include handling variations in lighting, poses, and facial expressions, as well as ensuring that the model generalizes well to various emotions and populations. This study introduces an automated facial emotion recognition using the pelican optimization algorithm with a deep convolutional neural network (AFER-POADCNN) model. The primary objective of the AFER-POADCNN model lies in the automatic recognition and classification of facial emotions. To accomplish this, the AFER-POADCNN model exploits the median-filtering (MF) approach to remove the noise present in it. Furthermore, the capsule-network (CapsNet) approach can be applied to the feature-extraction process, allowing the model to capture intricate facial expressions and nuances. To optimize the CapsNet model’s performance, hyperparameter tuning is undertaken with the aid of the pelican optimization algorithm (POA). This ensures that the model is finely tuned to detect a wide array of emotions and generalizes effectively across diverse populations and scenarios. Finally, the detection and classification of different kinds of facial emotions take place using a bidirectional long short-term memory (BiLSTM) network. The simulation analysis of the AFER-POADCNN system is tested on a benchmark FER dataset. The comparative result analysis showed the better performance of the AFER-POADCNN algorithm over existing models, with a maximum accuracy of 99.05%.

Facial expression recognition is a crucial task in numerous applications, including human-computer interaction, mental health monitoring, and human behavior analysis. Previous studies have primarily focused on individual models or techniques for improving emotion classification accuracy. However, a comparative analysis of different neural network architectures' performance for facial expression recognition is lacking. The main objective of this study is to compare the performance of Convolutional Neural Network (CNN) and Residual Network (ResNet) on the Fer2013 dataset. The author aims to analyze their behavior during the initial training phases and identify the architectural advantages and challenges associated with each model. Both models are trained using the same experimental setup to ensure a fair comparison. The models are trained using the Fer2013 dataset. The author employs a standard protocol for data preprocessing and augmentation. Results show that CNN achieves an accuracy of around 0.5 in the initial stages of training, which is significantly higher than ResNet's accuracy of 0.25. However, as training progresses, ResNet may outperform CNN because it has a more complicated structure that can capture more complex patterns. CNN exhibits superior performance during the initial training stage of the Fer2013 dataset. This reason may lie behind the fact that CNN has a simpler structure which makes it more sensitive on the basic features of the data.

This paper describes a computer-based training program for active learning of agent technology, expert systems, neural networks and case-based reasoning by undergraduate students using a simple agent framework. While many machine learning (ML) and artificial intelligence (AI) courses teach ML and AI concepts by means of programming assignments, these assignments have usually no connection to how the student will apply the newly obtained knowledge to previously unseen, real-world problems. The pedagogy that we adopted here is computer-based active learning: teams of students are presented with well-defined assignments aimed at building intelligent agents for person identification and recognition of facial expressions and emotions from video recordings of their faces. Classroom experience indicates that the students found the specified programming assignments highly motivating. Objective evaluation studies suggest that students learn much more effectively when a contextualized, collaborative, constructive, and reflective approach is used than when an orthodox, objectivist approach to teaching ML and AI techniques is used alone

Emotion recognition plays an indispensable role in human–machine interaction system. The process includes finding interesting facial regions in images and classifying them into one of seven classes: angry, disgust, fear, happy, neutral, sad, and surprise. Although many breakthroughs have been made in image classification, especially in facial expression recognition, this research area is still challenging in terms of wild sampling environment. In this paper, we used multi-level features in a convolutional neural network for facial expression recognition. Based on our observations, we introduced various network connections to improve the classification task. By combining the proposed network connections, our method achieved competitive results compared to state-of-the-art methods on the FER2013 dataset.

Facial Emotions Are Accurately Encoded in the Neural Signal of Those With Autism Spectrum Disorder: A Deep Learning Approach

Multiple sclerosis (MS) may be associated with impaired perception of facial emotions. However, emotion recognition mediated by bodily postures has never been examined in these patients. Moreover, several studies have suggested a relation between emotion recognition impairments and alexithymia. This is in line with the idea that the ability to recognize emotions requires the individuals to be able to understand their own emotions. Despite a deficit in emotion recognition has been observed in MS patients, the association between impaired emotion recognition and alexithymia has received little attention. The aim of this study was, first, to investigate MS patient's abilities to recognize emotions mediated by both facial and bodily expressions and, second, to examine whether any observed deficits in emotions recognition could be explained by the presence of alexithymia. Thirty patients with MS and 30 healthy matched controls performed experimental tasks assessing emotion discrimination and recognition of facial expressions and bodily postures. Moreover, they completed questionnaires evaluating alexithymia, depression, and fatigue. First, facial emotion recognition and, to a lesser extent, bodily emotion recognition can be impaired in MS patients. In particular, patients with higher disability showed an impairment in emotion recognition compared with patients with lower disability and controls. Second, their deficit in emotion recognition was not predicted by alexithymia. Instead, the disease's characteristics and the performance on some cognitive tasks significantly correlated with emotion recognition. Impaired facial emotion recognition is a cognitive signature of MS that is not dependent on alexithymia.