Real-time Facial Expression Recognition Research Articles

A Transfer Learning Approach for Facial Emotion Recognition Using a Deep Learning Model

The facial expression recognition (FER) system is the process of identifying the emotional state of a person. Emotion recognition from facial expressions is a rapidly growing area of research with numerous applications, including psychology, marketing, and human-computer interaction. This paper presents a novel approach to FER employing deep learning techniques, specifically leveraging the power of transfer learning methodology. To address the challenge of accurately recognizing different emotions, including angry, disgusted, afraid, happy, sad, surprised, and neutral, from facial expressions, a pre-trained model MobileNetV2 architecture has been fine-tuned for improving accuracy. It is a lightweight convolutional neural network architecture, specifically designed for efficient on-device inference. For evaluating the performance of the proposed FER model, the state-of-the-art FER-2013 dataset, along random images and video clips has been employed. Experimental results demonstrate that the proposed FER model attained a remarkable accuracy rate exceeding 99% when tested on diverse sets of random images and video clips. Moreover, the system achieved a notable accuracy of 61% when evaluated against the FER-2013 dataset. Overall, this approach presents a significant advancement in the field of real-time facial expression recognition using the MobileNetV2 architecture with FER2013 dataset, which may improve the quality of human-computer interactions.

Real-time Facial Expression Recognition to Enhance Emotional Intelligence in Autism

The ability to interpret emotions from facial expressions is essential in social interaction. However, these abilities are complex for individuals with autism spectrum disorder (ASD). This study aims to develop a real-time human facial expression recognition system on a mobile phone application to help individuals with ASD. The expression recognition model was developed using the FER-2013 dataset with the VGG-16 architecture. The experimental results show that the model obtained from the VGG-16 architecture can produce the best accuracy and average F1 scores with 0.91 and 0.91, respectively. We also experiment with three individuals with ASD, resulting in increasing interest in learning the expression.

Real-Time Facial Expression Recognition: Advances, Challenges, and Future Directions

Facial emotion recognition (FER) is the technology or process of identifying and interpreting human emotions based on the analysis of facial expressions. It involves using computer algorithms and machine learning techniques to detect and classify emotional states from images or videos of human faces. Further, FER plays a vital role in recognizing and understanding human emotions to better interpret someone’s feelings, intentions, and attitudes. In the present time, it is widely used in various fields such as healthcare, human–computer interaction, law enforcement, security, and beyond. FER has a wide range of practical applications across various industries including Emotion Monitoring, Adaptive Learning, and Virtual Assistants. This paper presents a comparative analysis of FER algorithms, focusing on deep learning approaches. The performance of different datasets, including FER2013, JAFFE, AffectNet, and Cohn–Kanade, is evaluated using convolutional neural networks (CNNs), deep face, attentional convolutional networks (ACNs), and deep belief networks (DBNs). Among the tested algorithms, DBNs outperformed other algorithms, reaching the highest accuracy of 98.82%. These results emphasize the effectiveness of deep learning techniques, particularly DBNs, in FER. Additionally, outlining the advantages and disadvantages of current research on facial emotion identification might direct future research efforts in the direction of the most profitable directions.

Retracted: Real Time Facial Expression Recognition for Online Lecture

Retracted: Real Time Facial Expression Recognition for Online Lecture

Emotional Analysis using Deep Learning

Emotions are mental states that accompany physiological changes in the face, resulting in facial expressions. Sympathy, anger, worry, joy, fright, and other significant emotions are a few examples. Facial expressions play a significant role in non-verbal communication because they encapsulate a person's emotions. There has been a great deal of research done on computer modelling of human emotions. Computer modelling of human emotions has been made possible by computer technology. However, it is still in its infancy. The authors attempted to overcome limitations and create new opportunities as well as gain a better understanding and implement this simple form of human interaction in proposed computer-aided world. It has been made possible to evaluate and interpret genuine facial expressions in real time thanks to new techniques for collecting facial expressions and quick, highresolution pictures. The FER (Facial Expression Recognition) method currently relies on motionless frames, which makes it very hard to recognize foreground from background in the absence of motion information. This study describes a real-time facial expression identification system that detects faces using HAAR cascading classification and classifies facial expressions using convolutional neural networks. The system utilizes a webcam to dynamically display emotion text and accurately categorizes seven major emotions, including anger, disgust, fear, happiness, sadness, surprise, and neutrality. Real-time facial expression recognition may be utilised in a number of real-world applications, including as airport security, trade, and medical monitoring.

Academic Emotion Classification Using FER: A Systematic Review

Facial emotion expressions are among the most potent, natural, and powerful means of human communication. Due to the COVID-19 pandemic, educational institutions worldwide are forced to switch rapidly to remote and online learning. Students are currently in an emergency state and must adapt to various and readily accessible learning methods, such as mobile learning applications or an e-learning system. A systematic literature review (SLR) is conducted to extract and synthesize information such as the emotion classifier used in the facial expression recognition (FER) system, the dataset used, the preprocessing technique applied, the feature extraction approach used, and the strength and limitation of the previous studies. Based on the search criteria, 701 publications were initially retrieved from five different digital databases, of which 48 studies have been chosen as primary studies for further analysis. Based on the findings of this study, the deep learning approach is the most frequently adopted approach in classifying student emotions during online learning. FER-2013 is the most commonly used FER dataset in FER studies, while DAiSEE is the most used academic emotion dataset. Moreover, support vector machine (SVM) is the conventional learning emotion classifier that is widely used in the FER systems, while convolutional neural network (CNN) is the most frequently used deep learning classifier. Next, it was found that the number of real-time FER systems is less than that of non-real-time FER systems. Finally, the top-1 accuracy of 94.6% was achieved by the long-term recurrent convolutional network on the academic emotion dataset, and the limitation is that it has low illumination and a lack of frontal pose.

SqueezExpNet: Dual-stage convolutional neural network for accurate facial expression recognition with attention mechanism

Facial expression recognition (FER) using a deep convolutional neural network (DCNN) is important and challenging. Although a substantial effort is made to increase FER accuracy through DCNN, previous studies are still not sufficiently generalisable for real-world applications. Traditional FER studies are mainly limited to controlled lab-posed frontal facial images, which lack the challenges of motion blur, head poses, occlusions, face deformations and lighting under uncontrolled conditions. In this work, we proposed a SqueezExpNet architecture that can take advantage of local and global facial information for a highly accurate FER system that can handle environmental variations. Our network was divided into two stages: a geometrical attention stage that possesses a SqueezeNet-like architecture to obtain local highlight information and a spatial texture stage comprising several squeezed and expanded layers to exploit high-level global features. In particular, we created a weighted mask of 3D face landmarks and used element-wise multiplication with a spatial feature in the first stage to draw attention to important local facial regions. Next, we input the face spatial image and its augmentations into the second stage of the network. Finally, like a classifier, a recurrent neural network was designed to collaborate the highlighted information from dual stages rather than simply using the SoftMax function, thereby aiding in overcoming the uncertainties. Experiments covering basic and compound FER tasks were performed using the three leading facial expression datasets. Our strategy outperformed the existing DCNN methods and achieved state-of-the-art results. The developed architecture, adopted research methodology and reported findings may find potential applications of real-time FER in surveillance, health and feedback systems.

A Novel Efficient Patient Monitoring FER System Using Optimal DL-Features

Automated Facial Expression Recognition (FER) serves as the backbone of patient monitoring systems, security, and surveillance systems. Real-time FER is a challenging task, due to the uncontrolled nature of the environment and poor quality of input frames. In this paper, a novel FER framework has been proposed for patient monitoring. Preprocessing is performed using contrast-limited adaptive enhancement and the dataset is balanced using augmentation. Two lightweight efficient Convolution Neural Network (CNN) models MobileNetV2 and Neural search Architecture Network Mobile (NasNetMobile) are trained, and feature vectors are extracted. The Whale Optimization Algorithm (WOA) is utilized to remove irrelevant features from these vectors. Finally, the optimized features are serially fused to pass them to the classifier. A comprehensive set of experiments were carried out for the evaluation of real-time image datasets FER-2013, MMA, and CK+ to report performance based on various metrics. Accuracy results show that the proposed model has achieved 82.5% accuracy and performed better in comparison to the state-of-the-art classification techniques in terms of accuracy. We would like to highlight that the proposed technique has achieved better accuracy by using 2.8 times lesser number of features.

Time efficient real time facial expression recognition with CNN and transfer learning

Time efficient real time facial expression recognition with CNN and transfer learning

Real-Time Facial Expression Recognition and Speech Tran-scripts over an on-premise Video Conference Application

Real-Time Facial Expression Recognition and Speech Tran-scripts over an on-premise Video Conference Application

Retraction Note to: Real-time facial expression recognition for affect identification using multi-dimensional SVM

Retraction Note to: Real-time facial expression recognition for affect identification using multi-dimensional SVM

Real-Time Facial Expression Recognition Using Deep Learning with Application in the Active Classroom Environment

The quality of a teaching method used in a classroom can be assessed by observing the facial expressions of students. To automate this, Facial Expression Recognition (FER) can be employed. Based on the recognized emotions of students, teachers can improve their lectures by determining which activities during the lecture evoke which emotions and how these emotions are related to the tasks solved by the students. Previous work mostly addresses the problem in the context of passive teaching, where teachers present while students listen and take notes, and usually in online courses. We take this a step further and develop predictive models that can classify emotions in the context of active teaching, specifically a robotics workshop, which is more challenging. The two best generalizing models (Inception-v3 and ResNet-34) on the test set were combined with the goal of real-time emotion prediction on videos of workshop participants solving eight tasks using an educational robot. As a proof of concept, we applied the models to the video data and analyzed the predicted emotions with regard to activities, tasks, and gender of the participants. Statistical analysis showed that female participants were more likely to show emotions in almost all activity types. In addition, for all activity types, the emotion of happiness was most likely regardless of gender. Finally, the activity type in which the analyzed emotions were the most frequent was programming. These results indicate that students’ facial expressions are related to the activities they are currently engaged in and contain valuable information for teachers about what they can improve in their teaching practice.

Real‐time facial expression recognition based on iterative transfer learning and efficient attention network

Real-time facial expression recognition is the basis for computers to understand human emotions and detect abnormalities in time. To effectively solve the problems of server overload and privacy information leakage, a real-time facial expression recognition method based on iterative transfer learning and efficient attention network (EAN) for edge resource-constrained scenes is proposed in this paper. Firstly, an EAN is designed with its parameter number and computation amount strictly limited by depth separable convolution and local channel attention mechanism. Then, the soft labels of facial expression data were obtained by EAN based on the idea of knowledge distillation, so as to provide more supervision information for the training process. Finally, an iterative transfer learning method of teacher-student (T-S) network was proposed; it refines the soft labels of the teacher network and further improves the recognition accuracy of the student network. The tests on the public datasets, FER2013 and RAF-DB, show that this method can significantly reduce the model complexity and achieve high recognition accuracy. Compared with other advanced methods, the proposed method strikes a good balance between complexity and accuracy, and well meets the real-time deployment requirements of facial expression recognition technology for edge resource-constrained scenes.

Design and Evaluation of a Hands-Free Video Game Controller for Individuals With Motor Impairments

Over the past few decades, video gaming has evolved at a tremendous rate although game input methods have been slower to change. Game input methods continue to rely on two-handed control of the joystick and D-pad or the keyboard and mouse for simultaneously controlling player movement and camera actions. Bi-manual input poses a significant play impediment to those with severe motor impairments. In this work, we propose and evaluate a hands-free game input control method that uses real-time facial expression recognition. Through our novel input method, our goal is to enable and empower individuals with neurological and neuromuscular diseases, who may lack hand muscle control, to be able to independently play video games. To evaluate the usability and acceptance of our system, we conducted a remote user study with eight severely motor-impaired individuals. Our results indicate high user satisfaction and greater preference for our input system with participants rating the input system as easy to learn. With this work, we aim to highlight that facial expression recognition can be a valuable input method.

Fostering Emotion Recognition in Children with Autism Spectrum Disorder

Facial expressions are of utmost importance in social interactions, allowing communicative prompts for a speaking turn and feedback. Nevertheless, not all have the ability to express themselves socially and emotionally in verbal and non-verbal communication. In particular, individuals with Autism Spectrum Disorder (ASD) are characterized by impairments in social communication, repetitive patterns of behaviour, and restricted activities or interests. In the literature, the use of robotic tools is reported to promote social interaction with children with ASD. The main goal of this work is to develop a system capable of automatic detecting emotions through facial expressions and interfacing them with a robotic platform (Zeno R50 Robokind® robotic platform, named ZECA) in order to allow social interaction with children with ASD. ZECA was used as a mediator in social communication activities. The experimental setup and methodology for a real-time facial expression (happiness, sadness, anger, surprise, fear, and neutral) recognition system was based on the Intel® RealSense™ 3D sensor and on facial features extraction and multiclass Support Vector Machine classifier. The results obtained allowed to infer that the proposed system is adequate in support sessions with children with ASD, giving a strong indication that it may be used in fostering emotion recognition and imitation skills.

Real-Time Facial Expression Recognition Based on Attention Mechanism

Real-Time Facial Expression Recognition Based on Attention Mechanism

Two-pathway attention network for real-time facial expression recognition

Two-pathway attention network for real-time facial expression recognition

RETRACTED ARTICLE: Facial expression recognition based on computer deep learning algorithm: taking cognitive acceptance of college students as an example

Because facial expressions are the most effective non-verbal communication method and can express emotions, mentalities and intentions well, their recognition and detection technologies have gradually attracted the attention of academia and industry. However, the recognition accuracy and robustness of traditional facial expression recognition algorithms are low, and the change of facial expressions is a dynamic process. Traditional algorithms are not effective in real-time facial expression recognition. This paper is based on the recognition and detection method of convolutional neural network algorithm. Consider applying convolutional neural networks to facial expression recognition and detection. Although there are many existing convolutional network models, none of them is suitable for the recognition of images in various fields. The recognition of facial expressions requires the creation of a network model suitable for expression recognition. This article is based on the LeNet-5 network and the network structure and the internal structure has been optimized and improved. Add batch normalization to solve the problem of network model over fitting caused by different features. Select maximum overlap pooling and average overlap pooling to reduce the amount of data while fully extracting expression features. Effectively improve the accuracy and robustness of recognition, and increase the tolerance to light, posture, and obstructions.

Efficient convolutional neural network with multi-kernel enhancement features for real-time facial expression recognition

Facial expressions are the most direct external manifestation of personal emotions. Different from other pattern recognition problems, the feature difference between facial expressions is smaller. The general methods are difficult to effectively characterize the feature difference, or their parameters are too large to realize real-time processing. This paper proposes a lightweight mobile architecture and a multi-kernel feature facial expression recognition network, which can take into account the speed and accuracy of real-time facial expression recognition. First, a multi-kernel convolution block is designed by using three depthwise separable convolution kernels of different sizes in parallel. The small and the large kernels can extract local details and edge contour information of facial expressions, respectively. Then, the multi-channel information is fused to obtain multi-kernel enhancement features to better describe the differences between facial expressions. Second, a "Channel Split" operation is performed on the input of the multi-kernel convolution block, which can avoid repeated extraction of invalid information and reduce the amount of parameters to one-third of the original. Finally, a lightweight multi-kernel feature expression recognition network is designed by alternately using multi-kernel convolution blocks and depthwise separable convolutions to further improve the feature representation ability. Experimental results show that the proposed network achieves high accuracy of 73.3 and 99.5% on FER-2013 and CK + datasets, respectively. Furthermore, it achieves a speed of 78 frames per second on 640 × 480 video. It is superior to other state-of-the-art methods in terms of speed and accuracy.

Dual integrated convolutional neural network for real-time facial expression recognition in the wild

Automatic recognition of facial expressions in the wild is a challenging problem and has drawn a lot of attention from the computer vision and pattern recognition community. Since their emergence, the deep learning techniques have proved their efficacy in facial expression recognition (FER) tasks. However, these techniques are parameter intensive, and thus, could not be deployed on resource-constrained embedded platforms for real-world applications. To mitigate these limitations of the deep learning inspired FER systems, in this paper, we present an efficient dual integrated convolution neural network (DICNN) model for the recognition of facial expressions in the wild in real-time, running on an embedded platform. The designed DICNN model with just 1.08M parameters and 5.40 MB memory storage size achieves optimal performance by maintaining a proper balance between recognition accuracy and computational efficiency. We evaluated the DICNN model on four FER benchmark datasets (FER2013, FERPlus, RAF-DB, and CKPlus) using different performance evaluation metrics, namely the recognition accuracy, precision, recall, and F1-score. Finally, to provide a portable solution with high throughput inference, we optimized the designed DICNN model using TensorRT SDK and deployed it on an Nvidia Xavier embedded platform. Comparative analysis results with the other state-of-the-art methods revealed the effectiveness of the designed FER system, which achieved competitive accuracy with multi-fold improvement in the execution speed.