Automatic Facial Expression Recognition System Research Articles

Modelling appearance variations in expressive and neutral face image for automatic facial expression recognition

AbstractIn automatic facial expression recognition (AFER) systems, modelling the spatio‐temporal feature information in a specific manner, coalescing, and its effective utilization is challenging. The state‐of‐the‐art studies have examined integrating multiple features to enhance the recognition rate of AFER systems. However, the feature variations between expressive and neutral face images are not fully explored to identify the expression class. The proposed research presents an innovative approach to AFER by modelling appearance variations in both expressive and neutral face images. The prominent contributions of the work are developing a novel and hybrid feature space by integrating the discriminative feature distribution derived from expressive and neutral face images; preserving the highly discriminative latent feature distribution using autoencoders. Local binary pattern (LBP) and histogram of oriented gradients (HOG) are the feature descriptors employed to derive the discriminative texture and shape information, respectively. The component‐based approach is employed, wherein the features are derived from the salient facial regions instead of the whole face. The three‐stage stacked deep convolutional autoencoder (SDCA) and multi‐class support vector machine (MSVM) are employed to address dimensionality reduction and classification, respectively. The efficacy of the proposed model is substantiated by empirical findings, which establish its superiority in terms of accuracy in AFER tasks on widely recognized benchmark datasets.

A Deep Quantum Convolutional Neural Network Based Facial Expression Recognition For Mental Health Analysis.

The purpose of this work is to analyze how new technologies can enhance clinical practice while also examining the physical traits of emotional expressiveness of face expression in a number of psychiatric illnesses. Hence, in this work, an automatic facial expression recognition system has been proposed that analyzes static, sequential, or video facial images from medical healthcare data to detect emotions in people's facial regions. The proposed method has been implemented in five steps. The first step is image preprocessing, where a facial region of interest has been segmented from the input image. The second component includes a classical deep feature representation and the quantum part that involves successive sets of quantum convolutional layers followed by random quantum variational circuits for feature learning. Here, the proposed system has attained a faster training approach using the proposed quantum convolutional neural network approach that takes [Formula: see text] time. In contrast, the classical convolutional neural network models have [Formula: see text] time. Additionally, some performance improvement techniques, such as image augmentation, fine-tuning, matrix normalization, and transfer learning methods, have been applied to the recognition system. Finally, the scores due to classical and quantum deep learning models are fused to improve the performance of the proposed method. Extensive experimentation with Karolinska-directed emotional faces (KDEF), Static Facial Expressions in the Wild (SFEW 2.0), and Facial Expression Recognition 2013 (FER-2013) benchmark databases and compared with other state-of-the-art methods that show the improvement of the proposed system.

Tracking and Detecting Depression Level using Facial Recognition and PEN&IQ Test

These Human facial expressions convey a lot of information visually rather than articulately. Facial expression recognition plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications including, but not limited to, human behaviour understanding, detection of mental disorders, and synthetic human expressions. Recognition of facial expression by computer with high recognition rate is still a challenging task. Two popular methods utilized mostly in the literature for the automatic FER systems are based on geometry and appearance. Facial Expression Recognition usually performed in four-stages consisting of pre-processing, face detection, feature extraction, and expression classification. The human face is an important part of an individual’s body and plays an important role in knowing the individual’s mood. The face is where a human expresses all his basic emotions. In the existing system, they examine the mental state manually by assessing them but have many disadvantages like we cannot predict any accurate solutions based on the assessment score because we might be not sure what kind of emotions the human user would be all time. To overcome this problem, a novel system is proposed to suggest an effective solution for predicting the mental state dynamically, we propose a hybrid architecture invoking facial based emotion sequence, PEN test and IQ test. By consistent monitoring of a human’s emotion and subjecting to PEN and IQ tests, the human’s mental state is routed. Combination of the above three techniques provides promising results for mental state and self-control

Development of Real-Time Emotion Recognition System Using Facial Expressions

This research presents a real-time emotion recognition system that combines human-friendly machine interaction with picture processing. For many years, facial detection has been available. Moving further, it is possible to simulate the emotions that people express on their faces and experience in their brains through the use of video, electric signals, or image forms. Since it is hard for computers to detect emotions from images or videos and a difficult task for the human eye, machine emotion detection requires a variety of image processing approaches for feature extraction. The approach proposed in this paper consists of two primary processes: facial expression recognition (FER) and face detection. The experimental investigation of facial emotion recognition is the main topic of this study. An emotion detection system's workflow consists of face detection, feature extraction, pre-processing, classification, and image acquisition. The emotion identification system uses the Haar cascade algorithm, an object detection algorithm, to recognize faces in an image or a real-time video, and the KNN Classifier for image classification in order to identify such emotions. Using the webcam to capture real-time photos, this system operates. The goal of this research is to develop an automatic facial expression recognition system that can recognize various emotions. Based on these studies, the system may be able to distinguish between a number of people who are fearful, furious, shocked, sad, or pleased, among other emotions.

A Bibliography Review on Facial Expression Recognition System by Conventional Approaches

The Facial Expression Recognition system is the process of identifying the emotional state of a person. It plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications but is not limited to, human behavior understanding, detection of mental disorders, and synthetic human expressions. Research in this field has drawn much attention given the continually increasing number of publications in the last 10 years. In this paper, bibliographic research was used and a comprehensive review of relevant publications was taken from the Scopus database. The research is defined as any research requiring information to be gathered from published materials. This bibliographic research gives descriptive information about a piece of work, such as author, title, date of publication, etc. The analysis also allows mapping of the scope and structure of discipline; locating the established collaboration patterns among countries. This analysis also identifies the current research interests and potential directions for future research.

Employing face reading technology to study the effects of the fake news' message sidedness on consumers' emotional valence and arousal

PurposeThis study aims to investigate how message sidedness affects the impact of fake news posted on social media on consumers' emotional responses.Design/methodology/approachThe study involves a face-tracking experiment in which 198 participants were exposed to different fake news messages concerning the COVID-19 vaccine. Specifically, participants were exposed to fake news using (1) a one-sided negative fake news message in which the message was entirely unfavorable and (2) a two-sided fake news message in which the negative message was mixed with favorable information. Noldus FaceReader 7, an automatic facial expression recognition system, was used to recognize participants' emotions as they read fake news. The authors sampled 17,450 observations of participants' emotional responses.FindingsThe results provide evidence of the significant influence of message sidedness on consumers' emotional valence and arousal. Specifically, two-sided fake news positively influences emotional valence, while one-sided fake news positively influences emotional arousal.Originality/valueThe current study demonstrates that research on fake news posted on social media may particularly benefit from insights regarding the potential but often overlooked importance of strategic design choices in fake news messages and their impact on consumers' emotional responses.

Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling

Facial expressions reflect people’s feelings, emotions, and motives, attracting researchers to develop a self-acting automatic facial expression recognition system. With the advances of deep learning frameworks for automatic facial expression recognition, the model complexity, limited training samples, and subtle micro facial muscle movements make the facial emotion expression system challenging. This research proposed a deep learning framework using fine-grained facial action unit detection to identify facial activity, behavior, and mood and recognize a person’s emotions based on these individual patterns. The proposed facial expression recognition system involves pre-processing, feature representation and normalization, hyper-parameter tuning, and classification. Here, two different convolutional neural network models have been introduced because of feature learning and representation, followed by classification. Various advanced feature representation methods, such as image augmentation, matrix normalization, fine-tuning, and transfer learning methods, have been applied to improve the performance of the proposed work. The proposed work’s performance and efficiency are evaluated under different approaches. The proposed work has been tested on standard Static Facial Expressions in the Wild, short name SFEW 1.0, SFEW 2.0, and Indian Movie Face (IMFDB) benchmark databases. The performances of the proposed system due to these databases are 48.15%, 80.34%, and 64.17%, respectively. The quantitative analysis of these results is compared with the standard existing state-of-the-art methods that show the proposed model outperforms the other competing methods.

Facial Expression Recognition using Robust Algorithm based on Modern Machine Learning Technique

Recently facial expression recognition has turned out to be an interesting field in research because of more demand for security and the advancement of mobile devices. Due to many serious incidents like terrorists’ attack, there arises more concern to develop the security systems mainly in certain places like airports and border crossings where identification and verification are mandatory. On the other hand, these surveillance systems aid to identify the missing person, even though it is based on robust facial expression recognition algorithms and on the developed database for facial expression recognition. However, the human faces are complex and multidimensional which make the facial gesture extraction to be very challenging. Obviously, in high secured applications facial expression recognition (FER) systems are mandatory to avoid incidents. In this paper, the automatic facial expression recognition system is developed based on the machine learning algorithms for classification. This research reveals the identification of FER for the ease of communication. Hybridization of Adaptive Kernel function based Extreme Learning Machine with Chicken Swarm Optimization (HAKELM-CSO) algorithm is introduced for identifying the accurate facial expression among the large database. In this work, an approach is developed by applying the machine learning techniques for the automated classification on the image region. The major purpose of this research work is to overcome the flaws of traditional algorithms and to improve the process of facial expression recognition which could be used in various applications.

Texture based feature extraction using symbol patterns for facial expression recognition.

Facial expressions can convey the internal emotions of a person within a certain scenario and play a major role in the social interaction of human beings. In automatic Facial Expression Recognition (FER) systems, the method applied for feature extraction plays a major role in determining the performance of a system. In this regard, by drawing inspiration from the Swastik symbol, three texture based feature descriptors named Symbol Patterns (SP1, SP2 and SP3) have been proposed for facial feature extraction. SP1 generates one pattern value by comparing eight pixels within a 33 neighborhood, whereas, SP2 and SP3 generates two pattern values each by comparing twelve and sixteen pixels within a 55 neighborhood respectively. In this work, the proposed Symbol Patterns (SP) have been evaluated with natural, fibonacci, odd, prime, squares and binary weights for determining the optimal recognition accuracy. The proposed SP methods have been tested on MUG, TFEID, CK+, KDEF, FER2013 and FERG datasets and the results from the experimental analysis demonstrated an improvement in the recognition accuracy when compared to the existing FER methods.

HUMAN FACIAL EMOTION RECOGNITION USING CNN

Human beings express emotions in everyday interactions. Understanding their emotions and knowing how to react to these expressions greatly enhances the interaction. An automatic Facial Expression Recognition system must try to solve the following problems: detection and location of faces ina cluttered scene, facial feature extraction, and facial expression classification. Knowing the user emotion, the system can adapt to the user. Facial expressions play an important role in recognition of emotions and are used in the process of non-verbal communication. They are very important in daily emotional communication, just next to the tone of voice. They are also an indicator of feelings, allowing a man to express an emotional state. The main motivation behind this project is to detect mental health of an individual.

Automatic Multiface Expression Recognition Using Convolutional Neural Network

The human facial expressions convey a lot of information visually. Facial expression recognition plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications in human behavior understanding, detection of mental disorders and synthetic human expressions. Recognition of facial expression by computer with high recognition rate is still a challenging task. Most of the methods utilized in the literature for the automatic facial expression recognition systems are based on geometry and appearance. Facial expression recognition is usually performed in four stages consisting of pre-processing, face detection, feature extraction, and expression classification. In this paper we applied various deep learning methods to classify the seven key human emotions: anger, disgust, fear, happiness, sadness, surprise and neutrality. The facial expression recognition system developed is experimentally evaluated with FER dataset and has resulted with good accuracy.

Improving multi-view facial expression recognition through two novel texture-based feature representations

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

An Automatic Facial Expression Recognition System Employing Convolutional Neural Network with Multi-strategy Gravitational Search Algorithm

Facial expression recognition (FER) plays a vital role in image processing according to the widespread development of human interactive applications. In the past few years, various researchers have focused on FER for implementing it in different applications. The existing system suffers from various complexities such as low accuracy, computational cost, and poor recognition performances. In this article, we proposed a novel concept to recognize facial expressions. The proposed work comprises three sections that are pre-processing, feature extraction, and classification. The pre-processing techniques remove the unwanted data from the original image and enhance the crucial details for further processing. The Convolutional Neural Network (CNN) is used for feature extraction. But, it yields lower performance in terms of feature extraction due to the shortage of hyperparameter tuning. Hence, the Multi-strategy Gravitational Search Algorithm (M-GSA) is utilized to extract the facial expression features from the eyebrow movement, nose, chin, and lip corner of the facial images. The facial expressions are classified via the Support Vector Machine (SVM) classifier. In this work, the top five facial expressions such as surprise, sad, happy, fear, and angry with three facial expression datasets such as FER-2013 dataset, CK+ dataset, and JAFFE dataset. Ultimately, the proposed method demonstrates better classification accuracy and recognition rates than different kinds of state-of-art methods.

Determining Subcategories of Facial Expressions for Improved Performance in Practical Applications

In the context of facial expression recognition (FER), this paper reviews the fundamental theories of emotions and further explains the key dimensions of a defined emotional space. The main contribution of this paper is to propose a set of novel categorization methods for facial expressions to be used in the design of an automatic FER system. This novel categorization enables the facial expression to be interpreted in a better way that and to be more effective in practical applications of automatic FER systems. In order to validate the feasibility of the proposed categorization methods, a set of experiments is reported which investigates and analyzes the influence that the novel categorization brings to a multi-view FER system.

A computational study on aging effect for facial expression recognition

This work uses newly introduced variations of the sparse representation-based classifier (SRC) to challenge the issue of automatic facial expression recognition (FER) with faces belonging to a wide span of ages. Since facial expression is one of the most powerful and immediate ways to disclose individuals? emotions and intentions, the study of emotional traits is an active research topic both in psychology and in engineering fields. To date, automatic FER systems work well with frontal and clean faces, but disturbance factors can dramatically decrease their performance. Aging is a critical disruption element, which is present in any real-world situation and which can finally be considered thanks to the recent introduction of new databases storing expressions over a lifespan. This study addresses the FER with aging challenge using sparse coding (SC) that represents the input signal as the linear combination of the columns of a dictionary. Dictionary learning (DL) is a subfield of SC that aims to learn from the training samples the best space capable of representing the query image. Focusing on one of the main challenges of SC, this work compares the performance of recently introduced DL algorithms. We run both a mixed-age experiment, where all faces are mixed, and a within-age experiment, where faces of young, middle-aged, and old actors are processed independently. We first work with the entire face and then we improve our initial performance using only discriminative patches of the face. Experimental results provide a fair comparison between the two recently developed DL techniques. Finally, the same algorithms are also tested on a database of expressive faces without the aging disturbance element, so as to evaluate DL algorithms' performance strictly on FER

Optimization in Feature Extraction schemes on Static Images to Improve the Performance of Automatic Facial Expression Recognition Systems

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

Fusion of transformed shallow features for facial expression recognition

Facial expression conveys important signs about the human affective state, cognitive activity, intention and personality. In fact, the automatic facial expression recognition systems are getting more interest year after year due to its wide range of applications in several interesting fields such as human computer/robot interaction, medical applications, animation and video gaming. In this study, the authors propose to combine between different descriptors features (histogram of oriented gradients, local phase quantisation and binarised statistical image features) after applying principal component analysis on each of them to recognise the six basic expressions and the neutral face from the static images. Their proposed fusion method has been tested on four popular databases which are: JAFFE, MMI, CASIA and CK+, using two different cross-validation schemes: subject independent and leave-one-subject-out. The obtained results show that their method outperforms both the raw features concatenation and state-of-the-art methods.

Human vision inspired feature extraction for facial expression recognition

Facial expression is a powerful way for human emotional communications. According to various applications, automatic facial expression recognition becomes an interesting problem for researchers in different areas. An automatic facial expression recognition system recognizes the expressed emotion in input facial image using several processing stages. Feature extraction is a vital step in facial expression recognition. One of the most widely used techniques for feature extraction in machine vision is utilizing Gabor filter which is sensitive to lines at various orientations. The disadvantage of Gabor filters is their computational cost and large feature vector length. Inspiring the human vision system and stimulations of complex cells, in this paper, firstly, facial image is convolved with Gabor filters. Then, the achieved convolution matrices are properly coded based on the maximum and minimum responses. Finally, the feature vector is obtained by calculating the histogram of these codes. The length of achieved histogram for 16 and 8 Gabor filters are 240 and 56, respectively, which is considerably less than keeping all Gabor responses. The proposed method is (person-independently) evaluated on four facial expression recognition datasets including CK+, SFEW, MMI, and RAF-DB. The experimental results show that the proposed method outperforms existing image texture descriptors in facial expression recognition in both controlled and uncontrolled images.

A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data.

Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system.

Automatic Facial Expression Recognition Using Combined Geometric Features

This study presents a geometric feature based automatic facial expression recognition system. The proposed system utilises the facial landmark points to determine the relative distances between the facial features in order to capture deformities caused by the movement of facial muscles due to different expressions. Three feature sets are generated by using landmark coordinates, relative distances between the facial points and a combination of both. Discriminating power of each feature set is determined by training different classification models for classifying an image into six basic emotions or neutral state. The proposed system is validated on two publically available facial expression databases. Experimental results show good accuracy of 95.5% for MUG database on the combined features by using ensemble neural network.