Facial Expression Recognition System Research Articles

Modelling appearance variations in expressive and neutral face image for automatic facial expression recognition

AbstractIn automatic facial expression recognition (AFER) systems, modelling the spatio‐temporal feature information in a specific manner, coalescing, and its effective utilization is challenging. The state‐of‐the‐art studies have examined integrating multiple features to enhance the recognition rate of AFER systems. However, the feature variations between expressive and neutral face images are not fully explored to identify the expression class. The proposed research presents an innovative approach to AFER by modelling appearance variations in both expressive and neutral face images. The prominent contributions of the work are developing a novel and hybrid feature space by integrating the discriminative feature distribution derived from expressive and neutral face images; preserving the highly discriminative latent feature distribution using autoencoders. Local binary pattern (LBP) and histogram of oriented gradients (HOG) are the feature descriptors employed to derive the discriminative texture and shape information, respectively. The component‐based approach is employed, wherein the features are derived from the salient facial regions instead of the whole face. The three‐stage stacked deep convolutional autoencoder (SDCA) and multi‐class support vector machine (MSVM) are employed to address dimensionality reduction and classification, respectively. The efficacy of the proposed model is substantiated by empirical findings, which establish its superiority in terms of accuracy in AFER tasks on widely recognized benchmark datasets.

Multi-CNN based logical reasoning system for facial expression recognition on small-sample datasets

Facial expression recognition has long been an area of great interest across a wide range of fields. Deep learning is commonly employed in facial expression recognition and demonstrates excellent performance in large-sample classification tasks. However, deep learning models often encounter challenges when confronted with small-sample expression classification problems, as they struggle to extract sufficient relevant features from limited data, resulting in subpar performance. This paper presents a novel approach called the Multi-CNN Logical Reasoning System, which is based on local area recognition and logical reasoning. It initiates the process by partitioning facial expression images into two distinct components: eye action and mouth action. Subsequently, it utilizes logical reasoning based on the inherent relationship between local actions and global expressions to facilitate facial expression recognition. Throughout the reasoning process, it not only incorporates manually curated knowledge but also acquires hidden knowledge from the raw data. Experimental results conducted on two small-sample datasets derived from the KDEF and RaFD datasets demonstrate that the proposed approach exhibits faster convergence and higher prediction accuracy when compared to classical deep learning-based algorithms.

Effective Facial Expression Recognition System Using Machine Learning

The co Facial expression recognition (FER) is a topic that has seen a lot of study in computer vision and machine learning. In recent years, deep learning techniques have shown remarkable progress on FER tasks. With this abstract, A Novel Is Advised By Us FER method that combines combined use of k-nearest neighbours and long short-term memory algorithms better efficiency and accurate facial expression recognition. The proposed system features two primary steps—feature extraction and classification—to get results. When extracting features, we extract features from the facial images using the Local Binary Patterns (LBP) algorithm. LBP is a simple yet powerful feature extraction technique that captures texture information from the image. In the classification stage, we use the KNN and LSTM algorithms for facial expression recognition. KNN is a simple and effective classification algorithm that finds the k closest to the given value neighbours to the test training-set-sample and assigning it to the class that is most frequent among its neighbours. However, KNN has limitations in handling temporal information. To address this limitation, we propose to use LSTM, which is a subclass of RNNs capable of capturing temporal relationships in time series data. The LSTM network takes as input the LBP features of a sequence of facial images and processes them through a series of LSTM cells to estimate the ultimate coding of the phrase. We examine the planned and system on two publicly available records: the CK+ and the Oulu-CASIA datasets. According on the experimental findings, the proposed system achieves performance at the cutting edge on both datasets. The proposed system performs better than other state-of-the-art methods, including those that use deep learning systems, quantitatively, in terms of F1-score and precision.In conclusion, the proposed FER system that combines KNN and LSTM algorithms achieves high accuracy and an F1 score in recognising facial expressions from sequences of images. This system can be used in many contexts, including human-computer interaction, emotion detection, and behaviour analysis.

UFace

Facial expression recognition (FER) is a crucial task for human-computer interaction and a multitude of multimedia applications that typically call for friendly, unobtrusive, ubiquitous, and even long-term monitoring. Achieving such a FER system meeting these multi-requirements faces critical challenges, mainly including the tiny irregular non-periodic deformation of emotion movements, high variability in facial positions and severe self-interference caused by users' own other behavior. In this work, we present UFace, a long-term, unobtrusive and reliable FER system for daily life using acoustic signals generated by a portable smartphone. We design an innovative network model with dual-stream input based on the attention mechanism, which can leverage distance-time profile features from various viewpoints to extract fine-grained emotion-related signal changes, thus enabling accurate identification of many kinds of expressions. Meanwhile, we propose effective mechanisms to deal with a series of interference issues during actual use. We implement UFace prototype with a daily-used smartphone and conduct extensive experiments in various real-world environments. The results demonstrate that UFace can successfully recognize 7 typical facial expressions with an average accuracy of 87.8% across 20 participants. Besides, the evaluation of different distances, angles, and interferences proves the great potential of the proposed system to be employed in practical scenarios.

One level deep convolutional neural network for facial key points detection

Facial landmark detection has a lot of applications in face recognition, face alignment, facial expression recognition, video surveillance and security systems. In the existing literature, there are multiple methods utilizing convolutional neural networks (CNNs) that address this problem in various ways. In many cases, the models use a tree-like structure of CNNs to achieve better results. This paper proposes a combination of three parallel deep convolutional neural networks (DCNNs) to estimate the accurate localization of each keypoint. The first one focuses on the whole face to outperform five points, including the eyes, nose, and mouth corners. The second one focuses on the eyes-nose parts to outperform three points, specifically the eyes and nose. The last one focuses on the nose-mouth parts to outperform three points, namely the nose and mouth corners. Further, we combine all outputs of the three DCNNs and take the average value of each detected key point as the final output. In the first step, we improvthe the parameter efficiency and accuracy of each DCNNs through a set of experiments using the labeled face parts in-the-wild database (LFPW) and the helen facial feature dataset (Helen). Then, we demonstrate that our approach yields more accurate estimations of facial key points than two state-of-the-art methods and commercial software in terms of accuracy.

Review on Smart Helmet for Accident Prevention Using IOT

Abstract: Detecting and preventing drunk driving is of paramount importance as it has the potential to reduce accidents caused by alcohol impairment. Ongoing research is dedicated to developing techniques that can identify alcohol levels leading to unconsciousness, impairing a person's ability to function, walk, and make sound judgments. This research primarily leverages the capabilities of electronics and automotive componentsand concepts. Various devices are being explored, suchas different sensors from the MQ series and facial expression recognition systems. Among these, the MQ-3 Sensor has shown promise in the field of electronics for detecting alcohol concentration in individuals. Long- term environmental implications of these drink and drive detection technologies are being meticulouslyexamined. This study aims to enhance our understanding and explore the commercial viability of these detection techniques in the realm of safety precautions

Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and Datasets

Recent technological developments have enabled computers to identify and categorize facial expressions to determine a person’s emotional state in an image or a video. This process, called “Facial Expression Recognition (FER)”, has become one of the most popular research areas in computer vision. In recent times, deep FER systems have primarily concentrated on addressing two significant challenges: the problem of overfitting due to limited training data availability, and the presence of expression-unrelated variations, including illumination, head pose, image resolution, and identity bias. In this paper, a comprehensive survey is provided on deep FER, encompassing algorithms and datasets that offer insights into these intrinsic problems. Initially, this paper presents a detailed timeline showcasing the evolution of methods and datasets in deep facial expression recognition (FER). This timeline illustrates the progression and development of the techniques and data resources used in FER. Then, a comprehensive review of FER methods is introduced, including the basic principles of FER (components such as preprocessing, feature extraction and classification, and methods, etc.) from the pro-deep learning era (traditional methods using handcrafted features, i.e., SVM and HOG, etc.) to the deep learning era. Moreover, a brief introduction is provided related to the benchmark datasets (there are two categories: controlled environments (lab) and uncontrolled environments (in the wild)) used to evaluate different FER methods and a comparison of different FER models. Existing deep neural networks and related training strategies designed for FER, based on static images and dynamic image sequences, are discussed. The remaining challenges and corresponding opportunities in FER and the future directions for designing robust deep FER systems are also pinpointed.

Emotion Recognition based on Facial Expression Identification using Deep Learning Algorithm for Automation Music Healing Application

Daily activities, particularly in the workplace or other societal settings, can cause stress and pressure for those who must manage them. This stress can lead to decreased performance and achievement in daily tasks, and in more severe cases, individuals may seek consultation with a psychiatrist to address their stress and pressure. This work presents a system for emotion identification based on face expression recognition system. The system processes the face expression image using Convolution Neural Network (CNN). The face expression image is extracted and modeled based on CNN method. The identification result sent to database which transferred to the Android application played the song based on emotion identification. The system is installed on an Android cell phone, making it flexible and portable. The system has achieved 70% and 80% accuracy in emotion detection during training and testing, respectively.

A Transfer Learning Approach for Facial Emotion Recognition Using a Deep Learning Model

The facial expression recognition (FER) system is the process of identifying the emotional state of a person. Emotion recognition from facial expressions is a rapidly growing area of research with numerous applications, including psychology, marketing, and human-computer interaction. This paper presents a novel approach to FER employing deep learning techniques, specifically leveraging the power of transfer learning methodology. To address the challenge of accurately recognizing different emotions, including angry, disgusted, afraid, happy, sad, surprised, and neutral, from facial expressions, a pre-trained model MobileNetV2 architecture has been fine-tuned for improving accuracy. It is a lightweight convolutional neural network architecture, specifically designed for efficient on-device inference. For evaluating the performance of the proposed FER model, the state-of-the-art FER-2013 dataset, along random images and video clips has been employed. Experimental results demonstrate that the proposed FER model attained a remarkable accuracy rate exceeding 99% when tested on diverse sets of random images and video clips. Moreover, the system achieved a notable accuracy of 61% when evaluated against the FER-2013 dataset. Overall, this approach presents a significant advancement in the field of real-time facial expression recognition using the MobileNetV2 architecture with FER2013 dataset, which may improve the quality of human-computer interactions.

Real-time Facial Expression Recognition to Enhance Emotional Intelligence in Autism

The ability to interpret emotions from facial expressions is essential in social interaction. However, these abilities are complex for individuals with autism spectrum disorder (ASD). This study aims to develop a real-time human facial expression recognition system on a mobile phone application to help individuals with ASD. The expression recognition model was developed using the FER-2013 dataset with the VGG-16 architecture. The experimental results show that the model obtained from the VGG-16 architecture can produce the best accuracy and average F1 scores with 0.91 and 0.91, respectively. We also experiment with three individuals with ASD, resulting in increasing interest in learning the expression.

A Deep Quantum Convolutional Neural Network Based Facial Expression Recognition For Mental Health Analysis.

The purpose of this work is to analyze how new technologies can enhance clinical practice while also examining the physical traits of emotional expressiveness of face expression in a number of psychiatric illnesses. Hence, in this work, an automatic facial expression recognition system has been proposed that analyzes static, sequential, or video facial images from medical healthcare data to detect emotions in people's facial regions. The proposed method has been implemented in five steps. The first step is image preprocessing, where a facial region of interest has been segmented from the input image. The second component includes a classical deep feature representation and the quantum part that involves successive sets of quantum convolutional layers followed by random quantum variational circuits for feature learning. Here, the proposed system has attained a faster training approach using the proposed quantum convolutional neural network approach that takes [Formula: see text] time. In contrast, the classical convolutional neural network models have [Formula: see text] time. Additionally, some performance improvement techniques, such as image augmentation, fine-tuning, matrix normalization, and transfer learning methods, have been applied to the recognition system. Finally, the scores due to classical and quantum deep learning models are fused to improve the performance of the proposed method. Extensive experimentation with Karolinska-directed emotional faces (KDEF), Static Facial Expressions in the Wild (SFEW 2.0), and Facial Expression Recognition 2013 (FER-2013) benchmark databases and compared with other state-of-the-art methods that show the improvement of the proposed system.

NA

NA

Deep Learning Based Human Emotion Exposure Detection Using Vocal and Countenance

The subtle expression of emotions on a person's face might give insight into the ideas running through that person's head. Today's society depends on being able to read facial emotions. Faces are a universal language that humans use to communicate a common and basic set of emotions. In a variety of settings and spheres of life, emotion recognition has benefits. For the purposes of safety and health, it is beneficial and essential. Additionally, it is essential for quickly and simply determining human emotions at a particular time without actually asking. Facial expression with non verbal cues are important for interpersonal interaction. The method of determining a person's emotional state is done using a facial expression recognition system. The technique of identifying different facets of a person's facial expressions is called emotion recognition. One of the most potent and challenging jobs in social communication is identifying human emotion from video frames, or facial expressions. This system compares the captured image to the training dataset that is stored in the database, and then displays the emotional state of the image. Recognizing automatically, the human emotions in pictures and videos will be possible with the help of an algorithm that carries out the identification, extracting and evaluating these facial expressions. These systems allow detection for emotions such as joy, rage, sadness, disgust, surprise, fear, neutral, and more. The applications for DL-based human emotion detection attempt to comprehend the significance of these features of body language and apply this understanding to various datasets and sources of information for the face emotion CNN pretrained module mobilenetv2 employed. This can assist in identifying and extracting facial expressions. These algorithms outperform numerous datasets of photos and videos in terms of accuracy.

Tracking and Detecting Depression Level using Facial Recognition and PEN&IQ Test

These Human facial expressions convey a lot of information visually rather than articulately. Facial expression recognition plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications including, but not limited to, human behaviour understanding, detection of mental disorders, and synthetic human expressions. Recognition of facial expression by computer with high recognition rate is still a challenging task. Two popular methods utilized mostly in the literature for the automatic FER systems are based on geometry and appearance. Facial Expression Recognition usually performed in four-stages consisting of pre-processing, face detection, feature extraction, and expression classification. The human face is an important part of an individual’s body and plays an important role in knowing the individual’s mood. The face is where a human expresses all his basic emotions. In the existing system, they examine the mental state manually by assessing them but have many disadvantages like we cannot predict any accurate solutions based on the assessment score because we might be not sure what kind of emotions the human user would be all time. To overcome this problem, a novel system is proposed to suggest an effective solution for predicting the mental state dynamically, we propose a hybrid architecture invoking facial based emotion sequence, PEN test and IQ test. By consistent monitoring of a human’s emotion and subjecting to PEN and IQ tests, the human’s mental state is routed. Combination of the above three techniques provides promising results for mental state and self-control

FaceTopoNet: Facial Expression Recognition Using Face Topology Learning

Prior work has shown that the order in which different components of the face are learned using a sequential learner can play an important role in the performance of facial expression recognition systems. We propose FaceTopoNet, an end- to-end deep model for facial expression recognition, which is capable of learning an effective tree topology of the face. Our model then traverses the learned tree to generate a sequence, which is then used to form an embedding to feed a sequential learner. The devised model adopts one stream for learning structure and one stream for learning texture. The structure stream focuses on the positions of the facial landmarks, while the main focus of the texture stream is on the patches around the landmarks to learn textural information. We then fuse the outputs of the two streams by utilizing an effective attention-based fusion strategy. We perform extensive experiments on four large- scale in-the-wild facial expression datasets - namely AffectNet, FER2013, ExpW, and RAF-DB - and one lab-controlled dataset (CK+) to evaluate our approach. FaceTopoNet achieves state-of- the-art performance on three of the five datasets and obtains competitive results on the other two datasets. We also perform rigorous ablation and sensitivity experiments to evaluate the impact of different components and parameters in our model. Lastly, we perform robustness experiments and demonstrate that FaceTopoNet is more robust against occlusions in comparison to other leading methods in the area.

Convolutional neural network combined with the attention mechanism for facial emotion recognition

Facial Emotion Recognition (FER) holds great importance in the fields of computer vision and machine learning. In this study, the aim is to improve the accuracy of facial expression recognition by incorporating attention mechanisms into Convolutional Neural Networks (CNN) with FER2013 dataset, which consists of grayscale images categorized into seven expressions. The combination of proposed CNN architecture and attention mechanisms is thoroughly elucidated, emphasizing the operations and interactions of their components. Additionally, the effectiveness of the new model is evaluated through experiments, comparing its performance with existing approaches in terms of accuracy. Besides, the results demonstrate that the CNN architecture with attention mechanisms outperforms the original CNN by achieving an improved accuracy rate of 69.07%, which is higher than 68.04% accuracy rate of original CNN. Moreover, the study further discusses the confusion matrix analysis, revealing the challenges faced in recognizing specific emotions due to limited training data and vague facial features. In the future, this study suggests addressing these limitations through data augmentation and to reduce the gap between training and testing accuracy. Overall, this research highlights the potential of attention mechanisms in enhancing facial expression recognition systems, paving the way for advanced applications in various domains.

An Innovative Exploration of Psychological Research Methods Driven by Artificial Intelligence: A Case Study Using Facial Expression Technique to Improve Teenagers’ Self Awareness

Incorporating artificial intelligence (AI) into psychological research can significantly advance AI methodologies and concurrently address inherent biases and inaccuracies in traditional psychological research reliant on manual data collection and analysis. This study introduces an innovative fusion of traditional facial expression recognition, a method aimed at alleviating the self-awareness of teenager students, with contemporary AI facial recognition technology. Through the integrated facial expression recognition system, the project aims to motivate students to identify their emotion, subsequently enhancing their psychological well-being and alleviating self-care.

Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition

As Facial Expression Recognition (FER) systems become integrated into our daily lives, these systems need to prioritise making <i>fair</i> decisions instead of only aiming at higher individual accuracy scores. From surveillance systems, to monitoring the mental and emotional health of individuals, these systems need to balance the <i>accuracy versus fairness</i> trade-off to make decisions that do not unjustly discriminate against specific under-represented demographic groups. Identifying <i>bias</i> as a critical problem in facial analysis systems, different methods have been proposed that aim to mitigate bias both at data and algorithmic levels. In this work, we propose the novel use of Continual Learning (CL), in particular, using Domain-Incremental Learning (Domain-IL) settings, as a potent bias mitigation method to enhance the <i>fairness</i> of Facial Expression Recognition (FER) systems. We compare different non-Continual Learning (CL)-based and CL-based methods for their <i>performance</i> and <i>fairness scores</i> on expression recognition and Action Unit (AU) detection tasks using two popular benchmarks, the RAF-DB and BP4D datasets, respectively. Our experimental results show that CL-based methods, on average, outperform other popular bias mitigation techniques on both <i>accuracy</i> and <i>fairness</i> metrics.

Development of Real-Time Emotion Recognition System Using Facial Expressions

This research presents a real-time emotion recognition system that combines human-friendly machine interaction with picture processing. For many years, facial detection has been available. Moving further, it is possible to simulate the emotions that people express on their faces and experience in their brains through the use of video, electric signals, or image forms. Since it is hard for computers to detect emotions from images or videos and a difficult task for the human eye, machine emotion detection requires a variety of image processing approaches for feature extraction. The approach proposed in this paper consists of two primary processes: facial expression recognition (FER) and face detection. The experimental investigation of facial emotion recognition is the main topic of this study. An emotion detection system's workflow consists of face detection, feature extraction, pre-processing, classification, and image acquisition. The emotion identification system uses the Haar cascade algorithm, an object detection algorithm, to recognize faces in an image or a real-time video, and the KNN Classifier for image classification in order to identify such emotions. Using the webcam to capture real-time photos, this system operates. The goal of this research is to develop an automatic facial expression recognition system that can recognize various emotions. Based on these studies, the system may be able to distinguish between a number of people who are fearful, furious, shocked, sad, or pleased, among other emotions.

Face and Facial Expressions Recognition System for Blind People Using ResNet50 Architecture and CNN

Many blind individuals have difficulties in recognizing people’s facial expression which may impact their social interaction. With the recognition, the blind individuals can accurately interpret and respond to the emotions. There is a lack in the existing application with the combination of face and facial expressions recognition. The blind individuals have to rely on multiple applications to accomplish the same task, making it difficult and time-consuming for them to use. The paper aims to recognize faces and facial expressions for blind individuals and provides feedback in real-time. Three face detection algorithms of Haar Cascade Classifier, Dlib, and RetinaFace are compared. Dlib is chosen to process with Histogram of Oriented Gradients (HOG) and Support Vector Machine (SVM). It loads the pre-trained model, computes the HOG features, slide the window scanning at different scales, classify the windows using the SVM classifier, generate bounding boxes, and applying non-maximum suppression. ResNet50 architecture is employed to recognize face and Convolutional Neural Networks (CNN) is applied to recognize facial expression. The training accuracy is 70% and validation accuracy is 60%.