Field Of Facial Expression Recognition Research Articles

Facial expression recognition for low resolution images using convolutional neural networks and denoising techniques

Background/Objectives: There is only limited research work is going on in the field of facial expression recognition on low resolution images. Mostly, all the images in the real world will be in low resolution and might also contain noise, so this study is to design a novel convolutional neural network model (FERConvNet), which can perform better on low resolution images. Methods: We proposed a model and then compared with state-of-art models on FER2013 dataset. There is no publicly available dataset, which contains low resolution images for facial expression recognition (Anger, Sad, Disgust, Happy, Surprise, Neutral, Fear), so we created a Low Resolution Facial Expression (LRFE) dataset, which contains more than 6000 images of seven types of facial expressions. The existing FER2013 dataset and LRFE dataset were used. These datasets were divided in the ratio 80:20 for training and testing and validation purpose. A HDM is proposed, which is a combination of Gaussian Filter, Bilateral Filter and Non local means denoising Filter. This hybrid denoising method helps us to increase the performance of the convolutional neural network. The proposed model was then compared with VGG16 and VGG19 models. Findings: The experimental results show that the proposed FERConvNet_HDM approach is effective than VGG16 and VGG19 in facial expression recognition on both FER2013 and LRFE dataset. The proposed FERConvNet_HDM approach achieved 85% accuracy on Fer2013 dataset, outperforming the VGG16 and VGG19 models, whose accuracies are 60% and 53% on Fer2013 dataset respectively. The same FERConvNet_HDM approach when applied on LRFE dataset achieved 95% accuracy. After analyzing the results, our FERConvNet_HDM approach performs better than VGG16 and VGG19 on both Fer2013 and LRFE dataset. Novelty/Applications: HDM with convolutional neural networks, helps in increasing the performance of convolutional neural networks in Facial expression recognition. Keywords: Facial expression recognition; facial emotion; convolutional neural network; deep learning; computer vision

Face recognition using fractional coefficients and discrete cosine transform tool

Face recognition is a computer vision application based on biometric information for automatic person identification or verification from image sequence or a video frame. In this context DCT is the easy technique to determine significant parameters. Until now the main object is selection of the coefficients to obtain the best recognition. Many techniques rely on premasking windows to discard the high and low coefficients to enhance performance. However, the problem resides in the shape and size of premask. To improve discriminator ability in discrete cosine transform domain, we used fractional coefficients of the transformed images with discrete cosine transform to limit the coefficients area for a better performance system. Then from the selected bands, we use the discrimination power analysis to search for the coefficients having the highest power to discriminate different classes from each other. Feature selection algorithm is a key issue in all pattern recognition system, in fact this algorithm is utilized to define features vector among several ones, where these features are selected according a specified discrimination criterion. Many classifiers are used to evaluate our approach like, support vector machine and random forests. The proposed approach is validated with Yale and ORL Face databases. Experimental results prove the sufficiency of this method in face and facial expression recognition field.

A novel feature separation model exchange-GAN for facial expression recognition

Currently, with the rapid development of deep learning, many breakthroughs have been made in the field of facial expression recognition (FER). However, according to our prior knowledge, facial images contain not only expression-related features but also some identity-related features, and the identity-related features vary from person to person which often have a negative influence on the FER process. It is one of the most important challenges in the field of FER. In this paper, a novel feature separation model exchange-GAN is proposed for the FER task, which can realize the separation of expression-related features and expression-independent features with high purity. And the FER method based on the exchange-GAN can overcome the interference of identity-related features to a large extent. First, the feature separation is achieved by the exchange-GAN through partial feature exchange and various constraints. Then we ignore the expression-independent features, and conduct FER only according to the expression-related features to alleviate the adverse effect of identity-related features. Finally, some experiments are conducted on three famous databases with the FER methods proposed in this paper. The experimental results show that the proposed FER method can alleviate the interference of identity-related information through feature separation by the exchange-GAN and achieve excellent performance for the objects that have not appeared in the training set. What’s more, our method can obtain very competitive FER accuracy on the three experimental databases.

LAUN Improved StarGAN for Facial Emotion Recognition

In the field of facial expression recognition, deep learning is extensively used. However, insufficient and unbalanced facial training data in available public databases is a major challenge for improving the expression recognition rate. Generative Adversarial Networks (GANs) can produce more one-to-one faces with different expressions, which can be used to enhance databases. StarGAN can perform one-to-many translations for multiple expressions. Compared with original GANs, StarGAN can increase the efficiency of sample generation. Nevertheless, there are some defects in essential areas of the generated face, such as the mouth and the fuzzy side face image generation. To address these limitations, we improved StarGAN to alleviate the defects of images generation by modifying the reconstruction loss and adding the Contextual loss. Meanwhile, we added the Attention U-Net to StarGAN's generator, replacing StarGAN's original generator. Therefore, we proposed the Contextual loss and Attention U-Net (LAUN) improved StarGAN. The U-shape structure and skip connection in Attention U-Net can effectively integrate the details and semantic features of images. The network's attention structure can pay attention to the essential areas of the human face. The experimental results demonstrate that the improved model can alleviate some flaws in the face generated by the original StarGAN. Therefore, it can generate person images with better quality with different poses and expressions. The experiments were conducted on the Karolinska Directed Emotional Faces database, and the accuracy of facial expression recognition is 95.97%, 2.19% higher than that by using StarGAN. Meanwhile, the experiments were carried out on the MMI Facial Expression Database, and the accuracy of expression is 98.30%, 1.21% higher than that by using StarGAN. Moreover, experiment results have better performance based on the LAUN improved StarGAN enhanced databases than those without enhancement.

Facial Expression Recognition Using Computer Vision: A Systematic Review

Emotion recognition has attracted major attention in numerous fields because of its relevant applications in the contemporary world: marketing, psychology, surveillance, and entertainment are some examples. It is possible to recognize an emotion through several ways; however, this paper focuses on facial expressions, presenting a systematic review on the matter. In addition, 112 papers published in ACM, IEEE, BASE and Springer between January 2006 and April 2019 regarding this topic were extensively reviewed. Their most used methods and algorithms will be firstly introduced and summarized for a better understanding, such as face detection, smoothing, Principal Component Analysis (PCA), Local Binary Patterns (LBP), Optical Flow (OF), Gabor filters, among others. This review identified a clear difficulty in translating the high facial expression recognition (FER) accuracy in controlled environments to uncontrolled and pose-variant environments. The future efforts in the FER field should be put into multimodal systems that are robust enough to face the adversities of real world scenarios. A thorough analysis on the research done on FER in Computer Vision based on the selected papers is presented. This review aims to not only become a reference for future research on emotion recognition, but also to provide an overview of the work done in this topic for potential readers.

Three convolutional neural network models for facial expression recognition in the wild

Facial expression recognition (FER) in the wild is a novel and challenging topic in the field of human emotion perception. Different kinds of convolutional neural network (CNN) approaches have been applied to this topic, but few of them ever considered what kind of architecture was better for the FER research. In this paper, we proposed three novel CNN models with different architectures. The first one is a shallow network, named the Light-CNN, which is a fully convolutional neural network consisting of six depthwise separable residual convolution modules to solve the problem of complex topology and over-fitting. The second one is a dual-branch CNN which extracts traditional LBP features and deep learning features in parallel. The third one is a pre-trained CNN which is designed by transfer learning technique to overcome the shortage of training samples. Extensive evaluations on three popular datasets (public CK+, multi-view BU-3DEF and FER2013 datasets) demonstrated that our models were competitive and representative in the field of FER in the wild research. We achieved significant better results with comparisons to plenty of state-of-the-art approaches. Moreover, we provided discussions on the effectiveness and practicability of CNNs with different feature types and architectures for FER in the wild as well.

Improved Real-Time Facial Expression Recognition Based on a Novel Balanced and Symmetric Local Gradient Coding.

In the field of Facial Expression Recognition (FER), traditional local texture coding methods have a low computational complexity, while providing a robust solution with respect to occlusion, illumination, and other factors. However, there is still need for improving the accuracy of these methods while maintaining their real-time nature and low computational complexity. In this paper, we propose a feature-based FER system with a novel local texture coding operator, named central symmetric local gradient coding (CS-LGC), to enhance the performance of real-time systems. It uses four different directional gradients on 5 × 5 grids, and the gradient is computed in the center-symmetric way. The averages of the gradients are used to reduce the sensitivity to noise. These characteristics lead to symmetric of features by the CS-LGC operator, thus providing a better generalization capability in comparison to existing local gradient coding (LGC) variants. The proposed system further transforms the extracted features into an eigen-space using a principal component analysis (PCA) for better representation and less computation; it estimates the intended classes by training an extreme learning machine. The recognition rate for the JAFFE database is 95.24%, whereas that for the CK+ database is 98.33%. The results show that the system has advantages over the existing local texture coding methods.

Graph-based dynamic ensemble pruning for facial expression recognition

Ensemble learning is an effective method to enhance the recognition accuracy of facial expressions. The performance of ensemble learning can be affected by many factors, such as the accuracy of the classifier pool’s component members and the diversity of classifier pool. Therefore, choosing the component members of ensemble learning reasonably can be helpful to maintain or enhance the recognition rate of facial expressions. In this paper, we propose a novel dynamic ensemble pruning method called graph-based dynamic ensemble pruning (GDEP) and apply it to the field of facial expression recognition. The GDEP’s main intension is to solve the problem that in the dynamic ensemble pruning methods, the classifier selection process is heavily sensitive to the membership in test sample’s neighborhood. Like all other dynamic ensemble pruning methods, GDEP can be divided into three steps: 1) Construct the neighborhood; 2) Evaluate the classifiers’ performance; 3) Form the selected classifier subset according to the classifiers’ capacity for recognizing a specific test image. And in order to achieve the GDEP’s intension, in the first step, this paper chooses neighborhood members more carefully by taking use of the statistics of classifiers’ behavior to characterize the intensity and similarity of emotions in data samples, and using the geodesic distance to calculate the data samples’ similarity. In the second step, GDEP builds the must-link and cannot-link graphs in the neighborhood to measure the classifiers’ performance and reduce the impact of inappropriate samples in the neighborhood. The experiments on the Fer2013, JAFFE and CK+ databases show the effectiveness of GDEP and demonstrate that it can compete with many state-of-art methods.

Meta-Analysis of the First Facial Expression Recognition Challenge

Automatic facial expression recognition has been an active topic in computer science for over two decades, in particular facial action coding system action unit (AU) detection and classification of a number of discrete emotion states from facial expressive imagery. Standardization and comparability have received some attention; for instance, there exist a number of commonly used facial expression databases. However, lack of a commonly accepted evaluation protocol and, typically, lack of sufficient details needed to reproduce the reported individual results make it difficult to compare systems. This, in turn, hinders the progress of the field. A periodical challenge in facial expression recognition would allow such a comparison on a level playing field. It would provide an insight on how far the field has come and would allow researchers to identify new goals, challenges, and targets. This paper presents a meta-analysis of the first such challenge in automatic recognition of facial expressions, held during the IEEE conference on Face and Gesture Recognition 2011. It details the challenge data, evaluation protocol, and the results attained in two subchallenges: AU detection and classification of facial expression imagery in terms of a number of discrete emotion categories. We also summarize the lessons learned and reflect on the future of the field of facial expression recognition in general and on possible future challenges in particular.