Expressional Images Research Articles

Disentangling Identity and Pose for Facial Expression Recognition

Facial expression recognition (FER) is a challenging problem because the expression component is always entangled with other irrelevant factors, such as identity and head pose. In this work, we propose an identity and pose disentangled facial expression recognition (IPD-FER) model to learn more discriminative feature representation. We regard the holistic facial representation as the combination of identity, pose and expression. These three components are encoded with different encoders. For identity encoder, a well pre-trained face recognition model is utilized and fixed during training, which alleviates the restriction on specific expression training data in previous works and makes the disentanglement practicable on in-the-wild datasets. At the same time, the pose and expression encoder are optimized with corresponding labels. Combining identity and pose feature, a neutral face of input individual should be generated by the decoder. When expression feature is added, the input image should be reconstructed. By comparing the difference between synthesized neutral and expressional images of the same individual, the expression component is further disentangled from identity and pose. Experimental results verify the effectiveness of our method on both lab-controlled and in-the-wild databases and we achieve state-of-the-art recognition performance.

Facial Expression Recognition With Two-Branch Disentangled Generative Adversarial Network

Facial Expression Recognition (FER) is a challenging task in computer vision as features extracted from expressional images are usually entangled with other facial attributes, e.g., poses or appearance variations, which are adverse to FER. To achieve a better FER performance, we propose a model named Two-branch Disentangled Generative Adversarial Network (TDGAN) for discriminative expression representation learning. Different from previous methods, TDGAN learns to disentangle expressional information from other unrelated facial attributes. To this end, we build the framework with two independent branches, which are specific for facial and expressional information processing respectively. Correspondingly, two discriminators are introduced to conduct identity and expression classification. By adversarial learning, TDGAN is able to transfer an expression to a given face. It simultaneously learns a discriminative representation that is disentangled from other facial attributes for each expression image, which is more effective for FER task. In addition, a self-supervised mechanism is proposed to improve representation learning, which enhances the power of disentangling. Quantitative and qualitative results in both in-the-lab and in-the-wild datasets demonstrate that TDGAN is competitive to the state-of-the-art methods.

Self‐adaptive weighted synthesised local directional pattern integrating with sparse autoencoder for expression recognition based on improved multiple kernel learning strategy

This study presents a novel method for solving facial expression recognition (FER) tasks which uses a self‐adaptive weighted synthesised local directional pattern (SW‐SLDP) descriptor integrating sparse autoencoder (SA) features based on improved multiple kernel learning (IMKL) strategy. The authors’ work includes three parts. Firstly, the authors propose a novel SW‐SLDP feature descriptor which divides the facial images into patches and extracts sub‐block features synthetically according to both distribution information and directional intensity contrast. Then self‐adaptive weights are assigned to each sub‐block feature according to the projection error between the expressional image and neutral image of each patch, which can highlight such areas containing more expressional texture information. Secondly, to extract a discriminative high‐level feature, they introduce SA for feature representation, which extracts the hidden layer representation including more comprehensive information. Finally, to combine the above two kinds of features, an IMKL strategy is developed by effectively integrating both soft margin learning and intrinsic local constraints, which is robust to noisy condition and thus improve the classification performance. Extensive experimental results indicate their model can achieve competitive or even better performance with existing representative FER methods.

Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition

Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. In this paper, we present a novel model, named Deep Attentive Multi-path Convolutional Neural Network (DAM-CNN), for FER. Different from most existing models, DAM-CNN can automatically locate expression-related regions in an expressional image and yield a robust image representation for FER. The proposed model contains two novel modules: an attention-based Salient Expressional Region Descriptor (SERD) and the Multi-Path Variation-Suppressing Network (MPVS-Net). SERD can adaptively estimate the importance of different image regions for FER task, while MPVS-Net disentangles expressional information from irrelevant variations. By jointly combining SERD and MPVS-Net, DAM-CNN is able to highlight expression-relevant features and generate a variation-robust representation for expression classification. Extensive experimental results on both constrained datasets (CK+, JAFFE, TFEID) and unconstrained datasets (SFEW, FER2013, BAUM-2i) demonstrate the effectiveness of our DAM-CNN model.

Handling missing labels and class imbalance challenges simultaneously for facial action unit recognition

Facial action unit (AU) recognition has attracted great attention because of the applications in a wide range of fields. Missing labels and class imbalance (CIB) are both challenges for facial action unit recognition. Missing labels means that there are only apart label assignments for training samples. CIB is observed from two perspectives: firstly, the number of positive AUs is much smaller than that of negative AUs for each expressional image; secondly, the rate of positive samples of different AUs are significantly different. Both missing labels and CIB lead to performance degradation in AU recognition. In this work, we propose to handle these two challenges in AU recognition simultaneously. Specifically, we formulate AU recognition with missing labels as a multi label learning with missing labels (MLML) problem, which handles the missing label challenge naturally. However, different from most existing MLML approaches which usually employ same features from whole image for all classes, we select the most related features for each AU. To handle the CIB challenge, we further introduce class cardinality bounds which constrain the number of positive AUs for each data instance, as well as the number of positive labels for each AU in the overall dataset. The class cardinality bounds serve as linear constraints for the objective function, which turns the optimization NP-hard. Thus we present convex approximation based on the Lovasz extension, which leads to a linear program that can be efficiently solved by the alternative direction method of multipliers (ADMM). Experimental results on both posed and spontaneous facial expression datasets demonstrate the superiority of the proposed method compared to state-of-the-art.

Facial Expression Recognition Using Hierarchical Features With Deep Comprehensive Multipatches Aggregation Convolutional Neural Networks

Facial expression recognition (FER) has long been a challenging task in computer vision. In this paper, we propose a novel method, named deep comprehensive multipatches aggregation convolutional neural networks (CNNs), to solve the FER problem. The proposed method is a deep-based framework, which mainly consists of two branches of the CNN. One branch extracts local features from image patches while the other extracts holistic features from the whole expressional image. In the model, local features depict expressional details and holistic features characterize the high-level semantic information of an expression. We aggregate both local and holistic features before making classification. These two types of hierarchical features represent expressions in different scales. Compared with most current methods with single type of feature, the model can represent expressions more comprehensively. Additionally, in the training stage, a novel pooling strategy named expressional transformation-invariant pooling is proposed for handling nuisance variations, such as rotations, noises, etc. Extensive experiments are conducted on the famous the Extended Cohn-Kanade (CK+) dataset and the Japanese Female Facial Expression (JAFFE) database expression datasets, where the recognition results obtained.

TRENDS OF FRENCH PERSONALISM BY THE EXAMPLE OF PORTRAITS OF J.-D. RACHETTE. SCULPTURAL PORTRAIT OF FRENCH ROSSIKA IN THE END OF THE XVIII - BEGINNING OF THE XIX CENTURIES

Ideas of personalism is one of the most characteristic features of the sculptural portrait of French Rossika. Personal image is an element of enlightenment ideals and aesthetic concepts of French art. An important place in portrait of French Rossika takes the question of expressional images of the individuality. J.-D. Rachette is a major French sculptor who has made a significant contribution to the heritage of Russian art. He opens the metaphysics of the space of the Russian social world in the classical artistic canons and in the modeling of traits of personality. French sculptural portrait assumes a certain canons of the description of public life, social designation of the person. Trends of personalism revealed the idea of the representation of individuality in society in the context of the construction of the secular ideals. Rachette conveys the idea of the personal life and human activities in the cultural space of Russia. Sculptor attached great importance to the intuitive perception of the personality characteristics model and the high level of performance.

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different directions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recognition and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes.

Facial expressional image synthesis controlled by emotional parameters

Facial expressional image synthesis is an important technique in human–computer interaction, but it is also a difficult task, especially to make various realistic expressive face images with a flexible control mechanism. In this paper, a novel parameter driven method of synthesizing realistic comprehensive expressional image is proposed. In this method a kernel-based bi-factor factorization model is adopted to decompose two influence factors, identity and expression, and their interaction from a small training database. Therefore the facial expressional images in the training database can be covered by corresponding identity and expression vectors and their interaction matrix. Comprehensive expression image can be manipulated by means of linear combination of these basis expression vectors in a flexible manner. In order to make this trained model be capable of producing realistic expressional images of any person outside the training set, expression ratio image (ERI) and relative shape description are joined into our model to enhance the expressive power of our method. Experimental results show that realistic facial expressional image can be synthesized successfully from only one picture of someone who is quite different from persons in the training database, and controlled by a parameter vector efficiently and effectively.