Real-time Facial Expression Recognition Research Articles

Dual integrated convolutional neural network for real-time facial expression recognition in the wild

Automatic recognition of facial expressions in the wild is a challenging problem and has drawn a lot of attention from the computer vision and pattern recognition community. Since their emergence, the deep learning techniques have proved their efficacy in facial expression recognition (FER) tasks. However, these techniques are parameter intensive, and thus, could not be deployed on resource-constrained embedded platforms for real-world applications. To mitigate these limitations of the deep learning inspired FER systems, in this paper, we present an efficient dual integrated convolution neural network (DICNN) model for the recognition of facial expressions in the wild in real-time, running on an embedded platform. The designed DICNN model with just 1.08M parameters and 5.40 MB memory storage size achieves optimal performance by maintaining a proper balance between recognition accuracy and computational efficiency. We evaluated the DICNN model on four FER benchmark datasets (FER2013, FERPlus, RAF-DB, and CKPlus) using different performance evaluation metrics, namely the recognition accuracy, precision, recall, and F1-score. Finally, to provide a portable solution with high throughput inference, we optimized the designed DICNN model using TensorRT SDK and deployed it on an Nvidia Xavier embedded platform. Comparative analysis results with the other state-of-the-art methods revealed the effectiveness of the designed FER system, which achieved competitive accuracy with multi-fold improvement in the execution speed.

Real-Time Facial Expression Recognition Based on Edge Computing

In recent years, many large-scale information systems in the Internet of Things (IoT) can be converted into interdependent sensor networks, such as smart cities, smart medical systems, and industrial Internet systems. The successful application of edge computing in the IoT will make our algorithms faster, more convenient, lower overall costs, providing better business practices, and enhance sustainability. Facial action unit (AU) detection recognizes facial expressions by analyzing cues about the movement of certain atomic muscles in the local facial area. According to the detected facial feature points, we could calculate the values of AU, and then use classification algorithms for emotion recognition. In edge devices, using optimized and custom algorithms to directly process the raw image data from each camera, the detected emotions can be more easily transmitted to the end-user. Due to the tremendous network overhead of transferring the facial action unit feature data, it poses challenges of a real-time facial expression recognition system being deployed in a distributed manner while running in production. Therefore, we designed a lightweight edge computing-based distributed system using Raspberry Pi tailed for this need, and we optimized the data transfer and components deployment. In the vicinity, the front-end and back-end processing modes are separated to reduce round-trip delay, thereby completing complex computing tasks and providing high-reliability, large-scale connection services. For IoT or smart city applications and services, they can be made into smart sensing systems that can be deployed anywhere with network connections.

Research on Sound Source Localization and Real-Time Facial Expression Recognition for Security Robot

Security robots are intelligent equipment used to perform security tasks, which can play a role in reducing the labor intensity and duty risk of police. In view of the shortcomings of existing police security robots in human-computer interaction and cooperation, the sound source localization and real-time facial expression recognition for the security robot is introduced in this paper. The auditory system of the robot obtains external sound analog signals through a microphone array, and performs corresponding digital signal processing on the collected sound signals. In order to strengthen the information interaction capability between the security robot and operators, the Kinect with deep data processing capabilities is introduced in this paper. The experimental results show that the system can analyze and recognize the position and status of the operators.

Real-time Facial Expression Recognition using 3D Appearance and Geometric Network for Public Security

Real-time Facial Expression Recognition using 3D Appearance and Geometric Network for Public Security

RETRACTED ARTICLE: Real-time facial expression recognition for affect identification using multi-dimensional SVM

Automated detection of different affect states for human beings using facial expressions has attracted an increasing level of research attention for high accuracy. In general, existing facial expressions databases have been used in automated affect state detection to achieve better efficiency. However, minimal real-time experiments data has been used for detecting affect state. To efficiently automate the affect recognition process, in this work, we proposed a new classifier framework and acquired experimentally real-time dataset for training and testing phase. The participants’ face images are captured and processed on the Region of Interest (ROI) to determine the feature points, and hence feature vectors are obtained. The facial feature vectors are given as an input to the proposed multiclass classifier, Multi-dimensional Support Vector Machine (MDSVM), which efficiently identifies and classifies the different affect state. The MDSVM is designed multi-dimensional, based on the two-dimensional valence-arousal emotional model. Thus, two SVMs are used, one in level-1, which classifies the input dataset into two classes based on valence, similarly the second SVM in level-2 classifies the input dataset into two categories based on arousal. The efficiency is improved by using the 8-cross-validation method. Thus, the methodology proposed in this work combines the advantage of reliable dataset acquired experimentally, significant 12 feature vectors obtained from facial expressions, prominently active multidimensional SVM, and 8-cross validation method. Experimental results illustrate the efficacy of the proposed system for accurately classifying the affect state into four categories, Relax, Sad, Angry, and Happy. The average accuracy obtained is 94.25% without k-fold cross-validation and 95.88% with 8-fold cross-validation.

Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM

Deep learning has received increasing attention in all fields and has made considerable progress in facial expression recognition (FER). Mainly, the conventional FER methods are trained for constrained datasets, which may not operate well for real-time images. Such real-time image sequences limit the accuracy and efficacy of the traditional system. In this work, the authors present a novel deep learning framework which combines convolutional neural network (CNN) with long short-term memory (LSTM) cell for real-time FER. The novel framework has three main aspects: (i) two different pre-processing techniques are employed to handle illumination variances and to preserve subtle edge information of each image; (ii) correspondingly, the pre-processed images are inputted to the two individual CNN architecture which extracts the spatial features very effectively; (iii) spatial feature maps from two individual CNN layers are fused and integrated with an LSTM layer which extracts temporal relations between the successive frames. They experimented the authors’ proposed method on three publically available FER databases and also with self-created database. With pre-processing, their proposed model achieves comparable and better results to the state-of-the-art.

Real-time facial expression recognition using smoothed deep neural network ensemble

Facial emotion recognition (FER) has been extensively researched over the past two decades due to its direct impact in the computer vision and affective robotics fields. However, the available datasets to train these models include often miss-labelled data due to the labellers bias that drives the model to learn incorrect features. In this paper, a facial emotion recognition system is proposed, addressing automatic face detection and facial expression recognition separately, the latter is performed by a set of only four deep convolutional neural network respect to an ensembling approach, while a label smoothing technique is applied to deal with the miss-labelled training data. The proposed system takes only 13.48 ms using a dedicated graphics processing unit (GPU) and 141.97 ms using a CPU to recognize facial emotions and reaches the current state-of-the-art performances regarding the challenging databases, FER2013, SFEW 2.0, and ExpW, giving recognition accuracies of 72.72%, 51.97%, and 71.82% respectively.

Facial Expression Recognition based on Convolution Neural Network and Orthogonal Neighbourhood Preserving Projection

Aiming at the problems of low recognition rate and poor real-time performance of traditional expression recognition methods, a real-time facial expression recognition model for a convolution neural network (CNN) combined with orthogonal neighbourhood preserving projection (ONPP) is proposed. First, the input image is subjected to a series of pre-processing steps. Then, we use ONPP to reduce the feature dimensions, while preserving global geometry as well as local neighbourhood relationships. Finally, we train and fine-tune the CNN on a massive facial expression database. The method is compared with other mainstream methods on a massive expression database, and experimental results show that the proposed method has higher recognition accuracy and good real-time performance compared to other mainstream methods.

Facial expression recognition based on convolution neural network and orthogonal neighbourhood preserving projection

Aiming at the problems of low recognition rate and poor real-time performance of traditional expression recognition methods, a real-time facial expression recognition model for a convolution neural network (CNN) combined with orthogonal neighbourhood preserving projection (ONPP) is proposed. First, the input image is subjected to a series of pre-processing steps. Then, we use ONPP to reduce the feature dimensions, while preserving global geometry as well as local neighbourhood relationships. Finally, we train and fine-tune the CNN on a massive facial expression database. The method is compared with other mainstream methods on a massive expression database, and experimental results show that the proposed method has higher recognition accuracy and good real-time performance compared to other mainstream methods.

PyFER: A Facial Expression Recognizer Based on Convolutional Neural Networks

Facial expression recognition (FER), one of the most trending research areas of the Human-Machine Interaction, is the task of detecting emotions by analyzing facial expressions and this analysis plays a critical role as it conveys the clearest information regarding the emotions of people. Despite the fact that the traditional machine learning algorithms produce high accuracies for similar tasks, they lack to detect emotions of faces, which are captured in a spontaneous manner ( a.k.a. “in the wild”) or in different poses or environmental conditions. In this article, a novel convolutional neural network architecture, namely, PyFER , is proposed to address the FER problem, of which the efficiency was revealed thanks to the experiments conducted on a widely-used benchmark dataset. According to the experimental results, the accuracy of PyFER was calculated to be as high as 96.3% on a de-facto standard dataset, namely, CK +, and all facial expressions, except for $happiness$ , were correctly detected by PyFER , which is encouraging for future studies. 16.67% of the images that actually represented the facial expression happiness were misdetected as the facial expression fear. The experimental results confirmed that the proposed neural network architecture is fast enough to be integrated into real-time FER applications as it was able to complete the analysis of a given photo for an average of 12.8 milliseconds, which is in the tolerable limit to latency for real-time applications.

Real Time Facial Expression Recognition System Based on Deep Learning

The automatic detection of facial expressions is an active research topic, since its wide fields of applications in human-computer interaction, games, security or education. However, the latest studies have been made in controlled laboratory environments, which is not according to real world scenarios. For that reason, a real time Facial Expression Recognition System (FERS) is proposed in this paper, in which a deep learning approach is applied to enhance the detection of six basic emotions: happiness, sadness, anger, disgust, fear and surprise in a real-time video streaming. This system is composed of three main components: face detection, face preparation and face expression classification. The results of proposed FERS achieve a 65% of accuracy, trained over 35558 face images..

A new pipeline for the recognition of universal expressions of multiple faces in a video sequence

Facial Expression Recognition (FER) is a crucial issue in human-machine interaction. It allows machines to act according to facial expression changes. However, acting in real time requires recognizing the expressions at video speed. Usually, the video speed differs from one device to another. However, one of the standard settings for shooting videos is 24 fps. This speed is considered as the low-end of what our brain can perceive as fluid video. From this perspective, to achieve a real-time FER the image analysis must be completed, strictly, in less than 0.042 second no matter how the background complexity is or how many faces exists in the scene. In this paper, a new pipeline has been proposed in order to recognize the fundamental facial expressions for more than one person in real world sequence videos. First, the pipeline takes as input a video and performs a face detection and tracking. Regions of Interest (ROI) are extracted from the detected face in order to extract the shape information when applying the Histogram of Oriented Gradient (HOG) descriptor. The number of features yield by HOG de-scriptor is reduced by means of a Linear Discriminant Analysis (LDA). Then, a deep data analysis was carried out, exploiting the pipeline, for the objective of setting up the LDA classifier. The analysis aimed at proving the suitability of the decision rule selected to separate the facial expression clusters in the LDA training phase. To conduct our analysis, we used ChonKanade(CK+) database and F-measure as an evaluation metric to calculate the average recognition rates. An automatic evaluation over time is proposed, where labelled videos is utilized to investigate the suitability of the pipeline in real world condition. The pipeline results showed that the use of HOG descriptor and the LDA gives a high recognition rate of 94.66%. It should be noted that the proposed pipeline achieves an average processing time of 0.018 second, without requiring any device that speeds up the processing.

A Deep Learning System for Recognizing Facial Expression in Real-Time

This article presents an image-based real-time facial expression recognition system that is able to recognize the facial expressions of several subjects on a webcam at the same time. Our proposed methodology combines a supervised transfer learning strategy and a joint supervision method with center loss, which is crucial for facial tasks. A newly proposed Convolutional Neural Network (CNN) model, MobileNet, which has both accuracy and speed, is deployed in both offline and in a real-time framework that enables fast and accurate real-time output. Evaluations towards two publicly available datasets, JAFFE and CK+, are carried out respectively. The JAFFE dataset reaches an accuracy of 95.24%, while an accuracy of 96.92% is achieved on the 6-class CK+ dataset, which contains only the last frames of image sequences. At last, the average run-time cost for the recognition of the real-time implementation is around 3.57ms/frame on a NVIDIA Quadro K4200 GPU.

Real-time facial expression recognition using local appearance-based descriptors

Real-time facial expression recognition using local appearance-based descriptors

Improved Real-Time Facial Expression Recognition Based on a Novel Balanced and Symmetric Local Gradient Coding.

In the field of Facial Expression Recognition (FER), traditional local texture coding methods have a low computational complexity, while providing a robust solution with respect to occlusion, illumination, and other factors. However, there is still need for improving the accuracy of these methods while maintaining their real-time nature and low computational complexity. In this paper, we propose a feature-based FER system with a novel local texture coding operator, named central symmetric local gradient coding (CS-LGC), to enhance the performance of real-time systems. It uses four different directional gradients on 5 × 5 grids, and the gradient is computed in the center-symmetric way. The averages of the gradients are used to reduce the sensitivity to noise. These characteristics lead to symmetric of features by the CS-LGC operator, thus providing a better generalization capability in comparison to existing local gradient coding (LGC) variants. The proposed system further transforms the extracted features into an eigen-space using a principal component analysis (PCA) for better representation and less computation; it estimates the intended classes by training an extreme learning machine. The recognition rate for the JAFFE database is 95.24%, whereas that for the CK+ database is 98.33%. The results show that the system has advantages over the existing local texture coding methods.

Real-time Facial Expression Recognition to Track Non-verbal Behaviors as Lie Indicators During Interview

During interview, a psychologist should pay attention to every gesture and response, both verbal and nonverbal language/behaviors, made by the client. Psychologist certainly has limitation in recognizing every gesture and response that indicates a lie, especially in interpreting nonverbal behaviors that usually occurs in a short time. In this research, a real time facial expression recognition is proposed to track nonverbal behaviors to help psychologist keep informed about the change of facial expression that indicate a lie. The method tracks eye gaze, wrinkles on the forehead, and false smile using combination of face detection and facial landmark recognition to find the facial features and image processing method to track the nonverbal behaviors in facial features. Every nonverbal behavior is recorded and logged according to the video timeline to assist the psychologist analyze the behavior of the client. The result of tracking nonverbal behaviors of face is accurate and expected to be useful assistant for the psychologists.

Information retrieval from facial expression using voting to assert exigency

In diurnal life, there are circumstances in which communication without words is required, emoticons or facial expressions are used to exchange virtual dialogues. This genre of communication completely injects into situation(emergency), where the sole way to interconnect is by performing some unusual activities, eye blinks or through some emotions. The paper proposes a Sagacious Information Recuperation Technique (SIRT) for sensing an emergency by capturing intelligent information from behavioral (emotions) and biological (eye blinks) changes. The proposed technique receives input from real-time human attributes: eye blinks and emotions. One of the parameter of proposed SIRT is further validated through an implementation of a real time facial expression recognition system(RTFER) that effectively identifies various emotions in real time scenario. LBP features collected from training dataset (cohn-kanade) are further classified using Support Vector Machine (SVM), Multilayer Perceptron(MP) and a Voting system involving SVM, MP and ADAB(ADABoost) classification algorithms. Experiment on Cohn-Kanade (CK) dataset illustrate that Voting system with multiple classifiers outperform methods using single descriptors. Receiver Operating Characteristics (ROC), area under the curve, sensitivity, fall-out, precision, and f-measure are used to show the performance of the system qualitatively and quantitatively. Accuracy with the proposed Voting system with reduced attribute dataset is increased to 81.82% in comparison with existing support vector machine (75.38%) and multilayer perceptron (78.41%).

A Study on the Psychological Analysis System Using Machine Learning

Real-time facial expression recognition and analysis technology is recently drawing attention in areas of computer vision, computer graphics, and HCI. Recognition of user’s emotion on the basis of video and voice is drawing particular interest. The technology may help managers of households or hospitals. In the present study, video and voice were converted into digital data through MATLAB by using PCA(Principal Component Analysis), LDA(Linear Discriminant Analysis), KNN(K Nearest Neighbor) algorithms to analyze emotions through machine learning. The manager of the psychological analysis counseling system may understand a user’s emotion in an smart phone environment. This system of the present study may help the manager to have a smooth conversation or develop a smooth relationship with a user on the basis of the provided psychological analysis results.

Facial Expression Emotion Detection for Real-Time Embedded Systems

Recently, real-time facial expression recognition has attracted more and more research. In this study, an automatic facial expression real-time system was built and tested. Firstly, the system and model were designed and tested on a MATLAB environment followed by a MATLAB Simulink environment that is capable of recognizing continuous facial expressions in real-time with a rate of 1 frame per second and that is implemented on a desktop PC. They have been evaluated in a public dataset, and the experimental results were promising. The dataset and labels used in this study were made from videos, which were recorded twice from five participants while watching a video. Secondly, in order to implement in real-time at a faster frame rate, the facial expression recognition system was built on the field-programmable gate array (FPGA). The camera sensor used in this work was a Digilent VmodCAM — stereo camera module. The model was built on the Atlys™ Spartan-6 FPGA development board. It can continuously perform emotional state recognition in real-time at a frame rate of 30. A graphical user interface was designed to display the participant’s video in real-time and two-dimensional predict labels of the emotion at the same time.

An online spatio-temporal tensor learning model for visual tracking and its applications to facial expression recognition

Abstract Robust visual tracking remains a technical challenge in real-world applications, as an object may involve many appearance variations. In existing tracking frameworks, objects in an image are often represented as vector observations, which discounts the 2-D intrinsic structure of the image. By considering an image in its actual form as a matrix, we construct the 3rd order tensor based object representation to preserve the spatial correlation within the 2-D image and fully exploit the useful temporal information. We perform incremental update of the object template using the N-mode SVD to model the appearance variations, which reduces the influence of template drifting and object occlusions. The proposed scheme efficiently learns a low-dimensional tensor representation through adaptively updating the eigenbasis of the tensor. Tensor based Bayesian inference in the particle filter framework is then utilized to realize tracking. We present the validation of the proposed tracking system by conducting the real-time facial expression recognition with video data and a live camera. Experiment evaluation on challenging benchmark image sequences undergoing appearance variations demonstrates the significance and effectiveness of the proposed algorithm.