Facial Expression Recognition System Research Articles

Performance Improving of ANN with Preprocessing Stage in Human Face Expression Recognition System

In many face recognition systems, the important part is face detection. The task of detecting face is complex due to its variability present across human faces including color, pose, expression, position, and orientation. So, by using various modeling techniques it is convenient to recognize various facial expressions. The system proposed consists of three phases, the facial expression database, pre-processing and classification. To simulate and assess recognition efficiency based on different variables (network composition, learning patterns and pre-processing), we present both the Japanese Female Facial Expression Database (JAFFE) and the Extended Cohn-Kanade Dataset (CK+). Comparative approaches of data preprocessing include face detection, translation, normalization of global contrast and histogram equalization. Significant results were obtained with 85.52 percent accuracy particularly in comparison with some other pre-processing phases and raw data in single pre-processing phases. The result indicates the ANN classifier representation produces a satisfactory result which reaches more accuracy.

Optimizing Deep Convolutional Neural Network for Facial Expression Recognition

Facial expression recognition (FER) systems have attracted much research interest in the area of Machine Learning. We designed a large, deep convolutional neural network to classify 40,000 images in the data-set into one of seven categories (disgust, fear, happy, angry, sad, neutral, surprise). In this project, we have designed deep learning Convolution Neural Network (CNN) for facial expression recognition and developed model in Theano and Caffe for training process. The proposed architecture achieves 61% accuracy. This work presents results of accelerated implementation of the CNN with graphic processing units (GPUs). Optimizing Deep CNN is to reduce training time for system.

Optimizing Deep Convolutional Neural Network for Facial Expression Recognition

Facial expression recognition (FER) systems have attracted much research interest in the area of Machine Learning. We designed a large, deep convolutional neural network to classify 40,000 images in the data-set into one of seven categories (disgust, fear, happy, angry, sad, neutral, surprise). In this project, we have designed deep learning Convolution Neural Network (CNN) for facial expression recognition and developed model in Theano and Caffe for training process. The proposed architecture achieves 61% accuracy. This work presents results of accelerated implementation of the CNN with graphic processing units (GPUs). Optimizing Deep CNN is to reduce training time for system.

A REVIEW ON FACIAL EXPRESSION RECOGNITION SYSTEMS

Facial expression is one of the most important form of non-verbal communication. Facial expressions emit the feelings of a person, and it allows judging that person by others. Some can understand facial expressions of underlying emotions to some extent, whereas many of us cannot. Facial Expression recognition (FER) system is a system to recognize expressions from a person’s face. It plays an important part in today’s world in fields of mental disease diagnosis, and human social/physiological interaction detection. Various methods of FER exist. This paper provides a summary of various processes involved in FER.

Questionnaires or Inner Feelings: Who Measures the Engagement Better?

This work proposes an innovative method for evaluating users’ engagement, combining the User Engagement Scale (UES) questionnaire and a facial expression recognition (FER) system, active research topics of increasing interest in the human–computer interaction domain (HCI). The subject of the study is a 3D simulator that reproduces a virtual FabLab in which users can approach and learn 3D modeling software and 3D printing. During the interaction with the virtual environment, a structured-light camera acquires the face of the participant in real-time, to catch its spontaneous reactions and compare them with the answers to the UES closed-ended questions. FER methods allow overcoming some intrinsic limits in the adoption of questioning methods, such as the non-sincerity of the interviewees and the lack of correspondence with facial expressions and body language. A convolutional neural network (CNN) has been trained on the Bosphorus database (DB) to perform expression recognition and the classification of the video frames in three classes of engagement (deactivation, average activation, and activation) according to the model of emotion developed by Russell. The results show that the two methodologies can be integrated to evaluate user engagement, to combine weighted answers and spontaneous reactions and to increase knowledge for the design of the new product or service.

Multi-EmoNet: A Novel Multi-Task Neural Network for Driver Emotion Recognition

Abstract Driver’s emotion affects driving safety Hu et al. (2013), therefore monitoring driver’s emotion could benefit road safety. However, the complex illumination conditions in a vehicle cockpit significantly challenge the effectiveness of camera-based facial expression recognition (FER) systems. To solve this problem, we proposed Multi-EmoNet, a novel multi-task neural network, to classify human facial expression under illumination variations and to restore noisy images. Our experiments demonstrate these two tasks are complementary and together facilitate better network representation learning. Our approach obtains significantly better classification accuracy on images with illumination variation compared to the baseline networks. More importantly, the proposed multi-task network is a general architecture that can be applied to any noise involved image classification problem.

Artificial Neural Network Based Ensemble Approach for Multicultural Facial Expressions Analysis

Facial expressions convey exhaustive information about human emotions and the most interactive way of social collaborations, despite differences in ethnicity, culture, and geography. Due to cultural differences, the variations in facial structure, facial appearance, and facial expression representation are the main challenges to the facial expression recognition system. These variations necessitate the need for multicultural facial expression analysis. This study presents several computational algorithms to handle these variations to get high expression recognition accuracy. We propose an artificial neural network-based ensemble classifier for multicultural facial expression analysis. The facial images from the Japanese female facial expression database, Taiwanese facial expression image database, and RadBoud faces database are combined to form a multi-culture facial expression dataset. The participants in the multicultural dataset originate from four ethnic regions including Japanese, Taiwanese, Caucasians, and Moroccans. Local binary pattern, uniform local binary pattern, and principal component analysis are applied for facial feature representation. Experimental results prove that facial expressions are innate and universal across all cultures with minor variations.

WiFace: Facial Expression Recognition Using Wi-Fi Signals

Facial expression is an important form of human nonverbal communication. Recognition of this nonverbal sign may enable developers to understand the feedback of the servers of smart devices or advertising. Existing approaches for facial expression recognition are mainly based on cameras or on-body sensors, which are either sensitive to lighting conditions or require users to wear cumbersome devices on their faces. In this paper, we propose a new facial expression recognition system based on Wi-Fi signals, named WiFace. Our key intuition is that facial muscle movements in different expressions will induce distinctive waveform patterns in the time-series of Channel State Information (CSI) in Wi-Fi signals. We develop a series of algorithms to process the CSI signals and extract the most representative waveform patterns for facial expression classification. We build a fully-functional prototype of WiFace using commercial off-the-shelf devices, which can recognize 6 typical facial expressions. We conduct extensive experiments to evaluate the performance of WiFace, and the experimental results show that the average recognition accuracy is 94.0%.

Human-Robots Interaction by Facial Expression Recognition

This paper presents a facial expressions recognition system to command both mobile and arm robot. The proposed system mainly consists of two modules: facial expressions recognition and robots command. The first module aims to extract the ROI (Region Of Interest like: mouth, eyes, eyebrow) using Gradient Vector Flow (GVF) snake segmentation and the Euclidian distance calculation (compatible with the MPEG-4 description of the six universal emotions). To preserve the temporal aspect of the processing from FEEDTUM database (video file), Time Delay Neural Network (TDNN) is used as classifier of the universal facial expressions such as happiness, sadness, surprise, anger, fear, disgust and neutral. While the second module, analyzes recognized facial expressions and translates them into a language to communicate with robots by establishing command law.

Facial Sentiment Analysis Using AI Techniques: State-of-the-Art, Taxonomies, and Challenges

With the advancements in machine and deep learning algorithms, the envision of various critical real-life applications in computer vision becomes possible. One of the applications is facial sentiment analysis. Deep learning has made facial expression recognition the most trending research fields in computer vision area. Recently, deep learning-based FER models have suffered from various technological issues like under-fitting or over-fitting. It is due to either insufficient training and expression data. Motivated from the above facts, this paper presents a systematic and comprehensive survey on current state-of-art Artificial Intelligence techniques (datasets and algorithms) that provide a solution to the aforementioned issues. It also presents a taxonomy of existing facial sentiment analysis strategies in brief. Then, this paper reviews the existing novel machine and deep learning networks proposed by researchers that are specifically designed for facial expression recognition based on static images and present their merits and demerits and summarized their approach. Finally, this paper also presents the open issues and research challenges for the design of a robust facial expression recognition system.

Illumination Invariant Facial Expression Recognition using Convolutional Neural Networks

In this work, we propose a prospective novel method to address illumination invariant system for facial expression recognition. Facial expressions are used to convey nonverbal visual information among humans. This also plays a vital role in human-machine interface modules that have invoked attention of many researchers. Earlier machine learning algorithms require complex feature extraction algorithms and are relying on the size and uniqueness of features related to the subjects. In this paper, a deep convolutional neural network is proposed for facial expression recognition and it is trained on two publicly available datasets such as JAFFE and Yale databases under different illumination conditions. Furthermore, transfer learning is used with pre-trained networks such as AlexNet and ResNet-101 trained on ImageNet database. Experimental results show that the designed network could recognize up to 30% variation in the illumination and it achieves an accuracy of 92%.

Facial Expression Recognition using SVM With CNN and Handcrafted Features

The facial expression recognition system is playing vital role in many organizations, institutes, shopping malls to know about their stakeholders’ need and mind set. It comes under the broad category of computer vision. Facial expression can easily explain the true intention of a person without any kind of conversation. The main objective of this work is to improve the performance of facial expression recognition in the benchmark datasets like CK+, JAFFE. In order to achieve the needed accuracy metrics, the convolution neural network was constructed to extract the facial expression features automatically and combined with the handcrafted features extracted using Histogram of Gradients (HoG) and Local Binary Pattern (LBP) methods. Linear Support Vector Machine (SVM) is built to predict the emotions using the combined features. The proposed method produces promising results as compared to the recent work in [1].This is mainly needed in the working environment, shopping malls and other public places to effectively understand the likeliness of the stakeholders at that moment.

Facial Expression Recognition using Ensemble Learning Technique

The most natural, influential and powerful way to communicate or convey a message is face expressions. In the field of computer engineering, facial expression recognition system, is helpful in areas like healthcare system, computer graphics, biometric devices, mobile phones, etc. Technologies such as virtual reality (VR) and augmented reality (AR) make use of facial expression recognition to implement a natural, friendly communication with humans. In this paper an approach for Facial Expression Recognition using Ensemble Learning Technique has been proposed. Ensemble methods use various learning algorithms to obtain good predictive performance that could be obtained from any of the basic learning algorithms alone. In the proposed method, initially the features are extracted from static images using color histograms. This process is done for all images gathered in the training dataset. The ensemble technique is then applied on the featured dataset in order to categorize a given image into one of the six emotions, happy, sad, fear, angry, disgust, and surprise. A satisfactory result has been obtained using static image dataset taken from kaggle and uci machine learning repository.

Facial Expression Recognition Based on Random Forest and Convolutional Neural Network

As an important part of emotion research, facial expression recognition is a necessary requirement in human–machine interface. Generally, a face expression recognition system includes face detection, feature extraction, and feature classification. Although great success has been made by the traditional machine learning methods, most of them have complex computational problems and lack the ability to extract comprehensive and abstract features. Deep learning-based methods can realize a higher recognition rate for facial expressions, but a large number of training samples and tuning parameters are needed, and the hardware requirement is very high. For the above problems, this paper proposes a method combining features that extracted by the convolutional neural network (CNN) with the C4.5 classifier to recognize facial expressions, which not only can address the incompleteness of handcrafted features but also can avoid the high hardware configuration in the deep learning model. Considering some problems of overfitting and weak generalization ability of the single classifier, random forest is applied in this paper. Meanwhile, this paper makes some improvements for C4.5 classifier and the traditional random forest in the process of experiments. A large number of experiments have proved the effectiveness and feasibility of the proposed method.

Real Time Facial Expression Recognition System Based on Deep Learning

The automatic detection of facial expressions is an active research topic, since its wide fields of applications in human-computer interaction, games, security or education. However, the latest studies have been made in controlled laboratory environments, which is not according to real world scenarios. For that reason, a real time Facial Expression Recognition System (FERS) is proposed in this paper, in which a deep learning approach is applied to enhance the detection of six basic emotions: happiness, sadness, anger, disgust, fear and surprise in a real-time video streaming. This system is composed of three main components: face detection, face preparation and face expression classification. The results of proposed FERS achieve a 65% of accuracy, trained over 35558 face images..

Fast and Cost Efficient Face Detection System with CNN Using Raspberry PI

Face detection is an important process when it comes to computer vision. It will serve as an input to a Facial expression and Face recognition system. Modern “C.C.T.V” cameras with face detection features are costly and only few are connected to the internet. This paper proposes a Face detection system which detects faces with a fusion of Convolutional neural network and Gabor Filter. Gabor filter is used to extract important facial features and Convolutional neural network is used to train the model. Model weights files are executed in Raspberry PI which is cost efficient. Raspberry pi is connected to cloud service which will alert the user with SMS and E-mail.

CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human–Robot Interaction

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

A computational study on aging effect for facial expression recognition

This work uses newly introduced variations of the sparse representation-based classifier (SRC) to challenge the issue of automatic facial expression recognition (FER) with faces belonging to a wide span of ages. Since facial expression is one of the most powerful and immediate ways to disclose individuals? emotions and intentions, the study of emotional traits is an active research topic both in psychology and in engineering fields. To date, automatic FER systems work well with frontal and clean faces, but disturbance factors can dramatically decrease their performance. Aging is a critical disruption element, which is present in any real-world situation and which can finally be considered thanks to the recent introduction of new databases storing expressions over a lifespan. This study addresses the FER with aging challenge using sparse coding (SC) that represents the input signal as the linear combination of the columns of a dictionary. Dictionary learning (DL) is a subfield of SC that aims to learn from the training samples the best space capable of representing the query image. Focusing on one of the main challenges of SC, this work compares the performance of recently introduced DL algorithms. We run both a mixed-age experiment, where all faces are mixed, and a within-age experiment, where faces of young, middle-aged, and old actors are processed independently. We first work with the entire face and then we improve our initial performance using only discriminative patches of the face. Experimental results provide a fair comparison between the two recently developed DL techniques. Finally, the same algorithms are also tested on a database of expressive faces without the aging disturbance element, so as to evaluate DL algorithms' performance strictly on FER

Facial expression recognition for monitoring neurological disorders based on convolutional neural network.

Facial expressions are a significant part of non-verbal communication. Recognizing facial expressions of people with neurological disorders is essential because these people may have lost a significant amount of their verbal communication ability. Such an assessment requires time consuming examination involving medical personnel, which can be quite challenging and expensive. Automated facial expression recognition systems that are low-cost and noninvasive can help experts detect neurological disorders. In this study, an automated facial expression recognition system is developed using a novel deep learning approach. The architecture consists of four-stage networks. The first, second and third networks segment the facial components which are essential for facial expression recognition. Owing to the three networks, an iconize facial image is obtained. The fourth network classifies facial expressions using raw facial images and iconize facial images. This four-stage method combines holistic facial information with local part-based features to achieve more robust facial expression recognition. Preliminary experimental results achieved 94.44% accuracy for facial expression recognition on RaFD database. The proposed system produced 5% improvement than the facial expression recognition system by using raw images. This study presents a quantitative, objective and non-invasive facial expression recognition system to help in the monitoring and diagnosis of neurological disorders influencing facial expressions.

Optimization in Feature Extraction schemes on Static Images to Improve the Performance of Automatic Facial Expression Recognition Systems

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.