Reliable Facial Expression Research Articles

Ensemble-based face expression recognition approach for image sentiment analysis

<span>Sentiment analysis based on images is an evolving area of study. Developing a reliable facial expression recognition (FER) device remains a difficult challenge as recognizing emotional feelings reflected in an image is dependent on a diverse set of factors. This paper presented an ensemble-based model for FER that incorporates multiple classification models: i) customized convolutional neural network (CNN), ii) ResNet50, and iii) InceptionV3. The model averaging ensemble classifier method is used to ensemble the predictions from the three models. Subsequently, the proposed FER model is trained and tested on a dataset with an uncontrolled environment (FER-2013 dataset). The experiment demonstrated that ensembling multiple classifiers outperformed all single classifiers in classifying positive and neutral expressions (91.7%, 81.7% and 76.5% accuracy rate for happy, surprise, and neutral, respectively). However, when classifying disgust, anger, and sadness, the ResNet50 model alone is the better choice. Although the Custom CNN performs the best in classifying fear expression (55.7% accuracy), the proposed FER model can still classify fear expression with comparable performance (52.8% accuracy). This paper demonstrated the potential of using the ensemble-based method to enhance the performance of FER. As a result, the proposed FER model has shown a 72.3% accuracy rate.</span>

Subverting Privacy-Preserving GANs: Hiding Secrets in Sanitized Images

Unprecedented data collection and sharing have exacerbated privacy concerns and led to increasing interest in privacy-preserving tools that remove sensitive attributes from images while maintaining useful information for other tasks. Currently, state-of-the-art approaches use privacy-preserving generative adversarial networks (PP-GANs) for this purpose, for instance, to enable reliable facial expression recognition without leaking users' identity. However, PP-GANs do not offer formal proofs of privacy and instead rely on experimentally measuring information leakage using classification accuracy on the sensitive attributes of deep learning (DL)-based discriminators. In this work, we question the rigor of such checks by subverting existing privacy-preserving GANs for facial expression recognition. We show that it is possible to hide the sensitive identification data in the sanitized output images of such PP-GANs for later extraction, which can even allow for reconstruction of the entire input images, while satisfying privacy checks. We demonstrate our approach via a PP-GAN-based architecture and provide qualitative and quantitative evaluations using two public datasets. Our experimental results raise fundamental questions about the need for more rigorous privacy checks of PP-GANs, and we provide insights into the social impact of these.

Stabilized blendshape editing using localized Jacobian transpose descent

The blendshape approach is the predominant technique for creating high quality facial animation. Facial expressions are generated by altering weight parameters manually for each key-frame by using traditional slider interfaces. However, authoring production quality facial animation with this process requires time-consuming, labor intensive and iterative work for the artists. Direct manipulation interfaces address this problem with a pin-and-drag operation inspired by inverse kinematics approaches. Existing direct manipulation techniques generally make use of the pseudo-inverse of the matrix that contains all the blendshape target shape’s vertex positions. However, pseudo-inverse computation can give unexpected results during facial expression editing, and therefore leaves room for improvement. To this end, we propose an approach to reducing unexpected movements during blendshape weight editing based on using the transpose of the Jacobian matrix rather than the pseudo-inverse. Our approach extracts the deformation directions from the blendshape matrix, and directly maps the sparse constrained point movements to the extracted directions. Furthermore, we offer two additional complementary techniques that build on our proposed transposition algorithm: a hybrid approach that blends the pseudo-inverse algorithm with our transposition algorithm thereby obtaining the benefits of both methods, and a technique that confines the edits to the local region within a geodesic radius on the surface. Our experiments show that, unlike pseudo-inverse based formulations, our “transposition” algorithm results in smoother and more reliable facial expressions during the weight editing process. The proposed approach improves the fidelity of the generated facial expressions by keeping haphazard movements to a minimum. It is robust, efficient, easy to implement and operates on any blendshape model.

Accurate and Reliable Facial Expression Recognition Using Advanced Softmax Loss With Fixed Weights

An important challenge for facial expression recognition (FER) is that real-world training data are usually imbalanced. Although many deep learning approaches have been proposed to enhance the discriminative power of deep expression features and enable a good predictive effect, few works have focused on the multiclass imbalance problem. When supervised by softmax loss (SL), which is widely used in FER, the classifier is often biased against minority categories (i.e., smaller interclass angular distances). In this letter, we present advanced softmax loss (ASL) to mitigate the bias induced by data imbalance and hence increase accuracy and reliability. The proposed ASL essentially magnifies the interclass diversity in the angular space to enhance discriminative power in every category. The proposed loss can easily be implemented in any deep network. Extensive experiments on the FER2013 and real-world affective faces (RAF) databases demonstrate that ASL is significantly more accurate and reliable than many state-of-the-art approaches and that it can easily be plugged into other methods and improves their performance.

Design and Implementation of Deception Detection System Based on Reliable Facial Expression

Design and Implementation of Deception Detection System Based on Reliable Facial Expression

Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features

Accurate recognition of facial expression is a challenging problem especially from multi-scale and multi orientation face images. In this article, we propose a novel technique called Weber Local Binary Image Cosine Transform (WLBI-CT). WLBI-CT extracts and integrates the frequency components of images obtained through Weber local descriptor and local binary descriptor. These frequency components help in accurate classification of various facial expressions in the challenging domain of multi-scale and multi-orientation facial images. Identification of significant feature set plays a vital role in the success of any facial expression recognition system. Effect of multiple feature sets with varying block sizes has been investigated using different multi-scale images taken from well-known JAFEE, MMI and CK+ datasets. Extensive experimentation has been performed to demonstrate that the proposed technique outperforms the contemporary techniques in terms of recognition rate and computational time.

Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network

Emotional health plays very vital role to improve people’s quality of lives, especially for the elderly. Negative emotional states can lead to social or mental health problems. To cope with emotional health problems caused by negative emotions in daily life, we propose efficient facial expression recognition system to contribute in emotional healthcare system. Thus, facial expressions play a key role in our daily communications, and recent years have witnessed a great amount of research works for reliable facial expressions recognition (FER) systems. Therefore, facial expression evaluation or analysis from video information is very challenging and its accuracy depends on the extraction of robust features. In this paper, a unique feature extraction method is presented to extract distinguished features from the human face. For person independent expression recognition, depth video data is used as input to the system where in each frame, pixel intensities are distributed based on the distances to the camera. A novel robust feature extraction process is applied in this work which is named as local directional position pattern (LDPP). In LDPP, after extracting local directional strengths for each pixel such as applied in typical local directional pattern (LDP), top directional strength positions are considered in binary along with their strength sign bits. Considering top directional strength positions with strength signs in LDPP can differentiate edge pixels with bright as well as dark regions on their opposite sides by generating different patterns whereas typical LDP only considers directions representing the top strengths irrespective of their signs as well as position orders (i.e., directions with top strengths represent 1 and rest of them 0), which can generate the same patterns in this regard sometimes. Hence, LDP fails to distinguish edge pixels with opposite bright and dark regions in some cases which can be overcome by LDPP. Moreover, the LDPP capabilities are extended through principal component analysis (PCA) and generalized discriminant analysis (GDA) for better face characteristic illustration in expression. The proposed features are finally applied with deep belief network (DBN) for expression training and recognition.

Facial Expression Analysis for Content-Based Video Retrieval

In this work, we propose a technique for facial expression recognition to bridge the semantic gap among the features that can be extracted in a content-based video retrieval system. The paper aims to provide accurate and reliable facial expression recognition of a dominant person in video frames using deterministic binary cellular automata (DBCA). Both geometric and appearance-based features are used. Efficient dimension reduction techniques for face detection and recognition are applied. Using the facial action coding system (FACS), one can code automatically nearly any anatomically possible facial expression, deconstructing it into what are called as action units (AUs). By employing two-dimensional deterministic binary cellular automaton systems (2D-DBCA), a scheme is developed to classify the facial expressions representing various emotions to retrieve video scenes/shots. Extensive experiments on Cohn–Kanade database, Yale database, and large movie videos show the superiority of the proposed method, in comparison with support vector machines (SVMs), hidden Markov models (HMMs), and neural network (NN) classifiers.

3D Facial Landmark Tracking and Facial Expression Recognition

In this paper, we address the challenging computer vision problem of obtaining a reliable facial expression analysis from a naturally interacting person. We propose a system that combines a 3D generic face model, 3D head tracking, and 2D tracker to track facial landmarks and recognize expressions. First, we extract facial landmarks from a neutral frontal face, and then we deform a 3D generic face to fit the input face. Next, we use our real-time 3D head tracking module to track a person's head in 3D and predict facial landmark positions in 2D using the projection from the updated 3D face model. Finally, we use tracked 2D landmarks to update the 3D landmarks. This integrated tracking loop enables efficient tracking of the non-rigid parts of a face in the presence of large 3D head motion. We conducted experiments for facial expression recognition using both framebased and sequence-based approaches. Our method provides a 75.9% recognition rate in 8 subjects with 7 key expressions. Our approach provides a considerable step forward toward new applications including human-computer interactions, behavioral science, robotics, and game applications.

Robust Facial Expression Recognition Using Near Infrared Cameras

In human-human communication we use verbal, vocal and non-verbal signals to communicate with others. Facial expressions are a form of non-verbal communication, recognizing them helps to improve the human-machine interaction. This paper proposes a system for pose- and illumination-invariant recognition of facial expressions using near-infrared camera images and precise 3D shape registration. Precise 3D shape information of the human face can be computed by means of Constrained Local Models (CLM), which fits a dense model to an unseen image in an iterative manner. We used a multi-class SVM to classify the acquired 3D shape into different emotion categories. Results surpassed human performance and show poseinvariant performance. Varying lighting conditions can influence the fitting process and reduce the recognition precision. We built a near-infrared and visible light camera array to test the method with different illuminations. Results shows that the near-infrared camera configuration is suitable for robust and reliable facial expression recognition with changing lighting conditions.

Recognition of facial expressions using Gabor wavelets and learning vector quantization

Facial expression recognition has potential applications in different aspects of day-to-day life not yet realized due to absence of effective expression recognition techniques. This paper discusses the application of Gabor filter based feature extraction in combination with learning vector quantization (LVQ) for recognition of seven different facial expressions from still pictures of the human face. The results presented here are better in several aspects from earlier work in facial expression recognition. Firstly, it is observed that LVQ based feature classification technique proposed in this study performs better in recognizing fear expressions than multilayer perceptron (MLP) based classification technique used in earlier work. Secondly, this study indicates that the Japanese Female Facial Expression (JAFFE) database contains expressers that expressed expressions incorrectly and these incorrect images adversely affect the development of a reliable facial expression recognition system. By excluding the two expressers from the data set, an improvement in recognition rate from 87.51% to 90.22% has been achieved. The present study, therefore, proves the feasibility of computer vision based facial expression recognition for practical applications like surveillance and human computer interaction.