Facial Image Synthesis Research Articles

Construction of facial emotion database through subjective experiments and its application to deep learning-based facial image processing

As the development of interactive robots and machines, studies to understand and reproduce facial emotions by computers have become important research areas. For achieving this goal, several deep learning-based facial image analysis and synthesis techniques recently have been proposed. However, there are difficulties in the construction of facial image dataset having accurate emotion tags (annotations, metadata), because such emotion tags significantly depend on human perception and cognition. In this study, we constructed facial image dataset having accurate emotion tags through subjective experiments. First, based on image retrieval using the emotion terms, we collected more than 1,600,000 facial images from SNS. Next, based on a face detection image processing, we obtained approximately 380,000 facial region images as “big data.” Then, through subjective experiments, we manually checked the facial expression and the corresponding emotion tags of the facial regions. Finally, we achieved approximately 5,500 facial images having accurate emotion tags as “good data.” For validating our facial image dataset in deep learning-based facial image analysis and synthesis, we applied our dataset to CNN-based facial emotion recognition and GAN-based facial emotion reconstruction. Through these experiments, we confirmed the feasibility of our facial image dataset in deep learning-based emotion recognition and reconstruction.

Representation Learning by Rotating Your Faces.

The large pose discrepancy between two face images is one of the fundamental challenges in automatic face recognition. Conventional approaches to pose-invariant face recognition either perform face frontalization on, or learn a pose-invariant representation from, a non-frontal face image. We argue that it is more desirable to perform both tasks jointly to allow them to leverage each other. To this end, this paper proposes a Disentangled Representation learning-Generative Adversarial Network (DR-GAN) with three distinct novelties. First, the encoder-decoder structure of the generator enables DR-GAN to learn a representation that is both generative and discriminative, which can be used for face image synthesis and pose-invariant face recognition. Second, this representation is explicitly disentangled from other face variations such as pose, through the pose code provided to the decoder and pose estimation in the discriminator. Third, DR-GAN can take one or multiple images as the input, and generate one unified identity representation along with an arbitrary number of synthetic face images. Extensive quantitative and qualitative evaluation on a number of controlled and in-the-wild databases demonstrate the superiority of DR-GAN over the state of the art in both learning representations and rotating large-pose face images.

Principal component analysis for surface reflection components and structure in facial images and synthesis of facial images for various ages

In this paper, principal component analysis is applied to the distribution of pigmentation, surface reflectance, and landmarks in whole facial images to obtain feature values. The relationship between the obtained feature vectors and the age of the face is then estimated by multiple regression analysis so that facial images can be modulated for woman aged 10–70. In a previous study, we analyzed only the distribution of pigmentation, and the reproduced images appeared to be younger than the apparent age of the initial images. We believe that this happened because we did not modulate the facial structures and detailed surfaces, such as wrinkles. By considering landmarks and surface reflectance over the entire face, we were able to analyze the variation in the distributions of facial structures and fine asperity, and pigmentation. As a result, our method is able to appropriately modulate the appearance of a face so that it appears to be the correct age.

Open-set face recognition across look-alike faces in real-world scenarios

The open-set problem is among the problems that have significantly changed the performance of face recognition algorithms in real-world scenarios. Open-set operates under the supposition that not all the probes have a pair in the gallery. Most face recognition systems in real-world scenarios focus on handling pose, expression and illumination problems on face recognition. In addition to these challenges, when the number of subjects is increased for face recognition, these problems are intensified by look-alike faces for which there are two subjects with lower intra-class variations. In such challenges, the inter-class similarity is higher than the intra-class variation for these two subjects. In fact, these look-alike faces can be created as intrinsic, situation-based and also by facial plastic surgery. This work introduces three real-world open-set face recognition methods across facial plastic surgery changes and a look-alike face by 3D face reconstruction and sparse representation. Since some real-world databases for face recognition do not have multiple images per person in the gallery, with just one image per subject in the gallery, this paper proposes a novel idea to overcome this challenge by 3D modeling from gallery images and synthesizing them for generating several images. Accordingly, a 3D model is initially reconstructed from frontal face images in a real-world gallery. Then, each 3D reconstructed face in the gallery is synthesized to several possible views and a sparse dictionary is generated based on the synthesized face image for each person. Also, a likeness dictionary is defined and its optimization problem is solved by the proposed method. Finally, the face recognition is performed for open-set face recognition using three proposed representation classifications. Promising results are achieved for face recognition across plastic surgery and look-alike faces on three databases including the plastic surgery face, look-alike face and LFW databases compared to several state-of-the-art methods. Also, several real-world and open-set scenarios are performed to evaluate the proposed method on these databases in real-world scenarios.

A lighting robust fitting approach of 3D morphable model for face reconstruction

Three-dimensional morphable model (3DMM) is a powerful tool for recovering 3D shape and texture from a single facial image. The success of 3DMM relies on two things: an effective optimization strategy and a realistic approach to synthesizing face images. However, most previous methods have focused on developing an optimization strategy under Phong's synthesis approach. In this paper, we adopt a more realistic synthesis technique that fully considers illumination and reflectance in the 3DMM fitting process. Using the sphere harmonic illumination model (SHIM), our new synthesis approach can account for more lighting factors than Phong's model. Spatially varying specular reflectance is also introduced into the synthesis process. Under SHIM, the cost function is nearly linear for all parameters, which simplifies the optimization. We apply our new optimization algorithm to determine the shape and texture parameters simultaneously. The accuracy of the recovered shape and texture can be improved significantly by considering the spatially varying specular reflectance. Hence, our algorithm produces an enhanced shape and texture compared with previous SHIM-based methods that recover shape from feature points. Although we use just a single input image in a profile pose, our approach gives plausible results. Experiments on a well-known image database show that, compared to state-of-the-art methods based on Phong's model, the proposed approach enhances the robustness of the 3DMM fitting results under extreme lighting and profile pose.

Video-based face recognition and image synthesis from rotating head frames using nonlinear manifold learning by neural networks

This paper proposes a new video-based face recognition method that uses video frames of a subject rotating his/her head. In the experiment discussed here, the manifolds of video frames embedded in a high-dimensional video space were extracted using neural network (NN)-based models. This increased the recognition rate in comparison with a simple NN architecture (from 72.9 to 81.3 %). These models were inspired by manifold interpretations of the brain's visual perception. Next, the pose and person manifolds were separated using the neurons trained in the hidden or bottleneck layer of the network. Finally, the separated manifolds were used to synthesize face images at different angles from a single frontal image. Using video frames to extract these manifolds produces higher-quality images than using manifolds extracted from single images. This improvement in image quality was verified using the structural similarity index measure.

A Unified Regularization Framework for Virtual Frontal Face Image Synthesis

Taking advantage of the statistical learning-based point of view, several approaches of frontal face image synthesis have received remarkable achievement. However, the existing methods mainly utilize either ordinary least squares (OLS) or fixed <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">${l_1}$</tex> </formula> –norm penalized sparse regression to estimate the solution. For the former, the solution is unstable when the linear equations system is ill-conditioned. For the latter, sparsity is only considered, while the significance of local similarity between input image and each training sample is ignored. Thus the synthesized result fails to faithfully approximate the ground truth. Moreover, these traditional methods cannot ensure the consistency between corresponding patches in frontal and profile faces. To address these problems, we present a unified regularization framework (URF) by imposing two regularization terms onto the solution. Firstly, we introduce an <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">${l_2}$</tex></formula> -norm constraint and impose a diagonal weights matrix onto it, in which each diagonal entry is defined by the spatial distance between input image patch and individual patch in training set. Secondly, to mitigate the aforementioned inconsistency problem, we present a neighborhood consistency regularization term, motivated by manifold learning. Finally, we generalize our framework to the <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">${l_q}$</tex></formula> -norm penalized case. By adjusting the shrinkage parameter <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$q$</tex></formula> , the framework gets more flexibility to choose a reasonable sparse domain. Extensive experiments on CMU Multi-PIE database and CAS-PEAL-R1 database verify the efficacy of our method.

Synthesising frontal face image using elastic net penalty and neighbourhood consistency prior

Traditional frontal face image synthesis based on the l 1-penalty has achieved remarkable success. However, the l 1-penalty on reconstruction coefficients has the drawback of instability when processing high-dimensional data (e.g. a facial image including hundreds of pixels). Moreover, the traditional l 1-penalty-based method requires consistency between the corresponding patches in frontal and profile faces, which is hard to guarantee due to self-occlusion. To overcome the instability problem of the traditional method, an extension of the l 1-penalty-based frontal face synthesis method, which benefits from the nature of the elastic net, is presented. 3 addition, to enhance the aforementioned consistency, a neighbourhood consistency penalty is imposed onto the reconstruction coefficients using the connected neighbour patches of the current patch. Furthermore, to ensure the synthesised result faithfully approximates the ground truth, a sparse neighbour selection strategy is introduced for finding related neighbours adaptively. Experimental results demonstrate the superiority of the proposed method over some state-of-the-art methods in both visual and quantitative comparisons.

Evaluating Visibility of Age Spot and Freckle based on Analysis and Synthesis of Facial Color Image

Evaluating Visibility of Age Spot and Freckle based on Analysis and Synthesis of Facial Color Image

Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination.

CANDECOMP/PARAFAC (CP) tensor factorization of incomplete data is a powerful technique for tensor completion through explicitly capturing the multilinear latent factors. The existing CP algorithms require the tensor rank to be manually specified, however, the determination of tensor rank remains a challenging problem especially for CP rank . In addition, existing approaches do not take into account uncertainty information of latent factors, as well as missing entries. To address these issues, we formulate CP factorization using a hierarchical probabilistic model and employ a fully Bayesian treatment by incorporating a sparsity-inducing prior over multiple latent factors and the appropriate hyperpriors over all hyperparameters, resulting in automatic rank determination. To learn the model, we develop an efficient deterministic Bayesian inference algorithm, which scales linearly with data size. Our method is characterized as a tuning parameter-free approach, which can effectively infer underlying multilinear factors with a low-rank constraint, while also providing predictive distributions over missing entries. Extensive simulations on synthetic data illustrate the intrinsic capability of our method to recover the ground-truth of CP rank and prevent the overfitting problem, even when a large amount of entries are missing. Moreover, the results from real-world applications, including image inpainting and facial image synthesis, demonstrate that our method outperforms state-of-the-art approaches for both tensor factorization and tensor completion in terms of predictive performance.

“Wow! You Are So Beautiful Today!”

Beauty e-Experts, a fully automatic system for makeover recommendation and synthesis, is developed in this work. The makeover recommendation and synthesis system simultaneously considers many kinds of makeover items on hairstyle and makeup. Given a user-provided frontal face image with short/bound hair and no/light makeup, the Beauty e-Experts system not only recommends the most suitable hairdo and makeup, but also synthesizes the virtual hairdo and makeup effects. To acquire enough knowledge for beauty modeling, we built the Beauty e-Experts Database, which contains 1,505 female photos with a variety of attributes annotated with different discrete values. We organize these attributes into two different categories, beauty attributes and beauty-related attributes. Beauty attributes refer to those values that are changeable during the makeover process and thus need to be recommended by the system. Beauty-related attributes are those values that cannot be changed during the makeup process but can help the system to perform recommendation. Based on this Beauty e-Experts Dataset, two problems are addressed for the Beauty e-Experts system: what to recommend and how to wear it, which describes a similar process of selecting hairstyle and cosmetics in daily life. For the what-to-recommend problem, we propose a multiple tree-structured supergraph model to explore the complex relationships among high-level beauty attributes, mid-level beauty-related attributes, and low-level image features. Based on this model, the most compatible beauty attributes for a given facial image can be efficiently inferred. For the how-to-wear-it problem, an effective and efficient facial image synthesis module is designed to seamlessly synthesize the recommended makeovers into the user facial image. We have conducted extensive experiments on testing images of various conditions to evaluate and analyze the proposed system. The experimental results well demonstrate the effectiveness and efficiency of the proposed system.

Sparse frontal face image synthesis from an arbitrary profile image

Frontal face image synthesis from an arbitrary profile image plays an important role in automatic video surveillance systems, and remains a challenge in computer vision. The strategies of partition are popular and promising for synthesizing frontal face images. However, conventional rectangular partition criterions fail to align corresponding patches in profile images and frontal face images. Given an arbitrary profile image, to synthesize a corresponding frontal face image which is smooth in texture and similar in appearance, we introduce a triangulation-based partition criterion and do synthesis based on sparse representation. The triangulation-based partition ensures the corresponding triangular patches are strictly aligned. And sparse representation adaptively finds the most similar patches for synthesis while abandons unlike patches. Furthermore, a confederate learning strategy is proposed to reduce the blocking artifacts caused by triangulation-based partition. Experimental results conducted on the Oriental Face database demonstrate the effectiveness of the proposed frontal face image synthesis method and advantages over previous works.

Pose-robust face recognition via sparse representation

We propose a pose-robust face recognition method to handle the challenging task of face recognition in the presence of large pose difference between gallery and probe faces. The proposed method exploits the sparse property of the representation coefficients of a face image over its corresponding view-dictionary. By assuming the representation coefficients are invariant to pose, we can synthesize for the probe image a novel face image which has smaller pose difference with the gallery faces. Furthermore, face recognition in the presence of pose variations is achieved based on the synthesized face image again via sparse representation. Extensive experiments on CMU Multi-PIE face database are conducted to verify the efficacy of the proposed method.

One-to-many neural network mapping techniques for face image synthesis

This paper investigates the performance of neural network-based techniques applied to the problem of defining the relationship between a particular type of variation in face images and the multivariate data distributions of these images. In this respect the problem of defining a mapping associating a quantified facial attribute and the overall typical facial appearance is addressed. In particular the applicability of formulating a mapping function using neural network-based methods like Multilayer Perceptrons (MLPs), Radial Basis Functions (RBFs), Mixture Density Networks (MDNs) and a latent variable method, the General Topographic Mapping (GTM) is investigated. Quantitative and visual results obtained during the experimental investigation, suggest that for one-to-many problems, where the entire variance of the distribution is not required, the RBFs are the best options when compared to MLPs, MDNs and GTM. The proposed techniques can be applied to applications involving face image synthesis and other applications that require one-to-many mapping transformations.

Synthesis of facial image with expression based on muscular contraction parameters using linear muscle and sphincter muscle

AbstractWe aim to synthesize individual facial images with expression based on muscular contraction parameters. We have proposed a method of calculating the muscular contraction parameters from an arbitrary face image without using learning for each individual. As a result, we were able to generate not only individual facial expressions, but also the facial expressions of various persons. In this paper, we propose a muscle‐based facial model, in which the facial muscles defined are both linear muscles and sphincter muscles. Additionally, we propose a method of synthesizing individual facial images with expression based on muscular contraction parameters. First, an individual facial model with expression is generated by fitting using an arbitrary face image. Next, the muscular contraction parameters that correspond to the expression displacement of the input face image are calculated. Finally, the facial expression is synthesized by the vertex displacements of a neutral facial model based on the calculated muscular contraction parameters. Experimental results reveal that the newly included sphincter muscles make it possible to synthesize facial expressions of the facial image which correspond to the actual face image with arbitrary expressions of the mouth or eyes. © 2011 Wiley Periodicals, Inc. Electron Comm Jpn, 94(10): 17–25, 2011; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.10362

Facial Image Synthesis Considering Illumination Variations on Mobile Devices

Facial Image Synthesis Considering Illumination Variations on Mobile Devices

Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis [Social Sciences

The face conveys information about a person's age, sex, background, and identity; what they are feeling, thinking, or likely to do next. Facial expression regulates face-to-face interactions, indicates reciprocity and interpersonal attraction or repulsion, and enables intersubjectivity between members of different cultures. Facial expression indexes neurological and psychiatric functioning and reveals personality and socioemotional development. Not surprisingly, the face has been of keen interest to behavioral scientists. About 15 years ago, computer scientists became increasingly interested in the use of computer vision and graphics to automatically analyze and synthesize facial expression. This effort was made possible in part by the development in psychology of detailed coding systems for describing facial actions and their relation to basic emotions, that is, emotions that are interpreted similarly in diverse cultures. The most detailed of these systems, the Facial Action Coding System (FACS), informed the development of the MPEG-4 facial animation parameters for video transmission and enabled progress toward automated measurement and synthesis of facial actions for research in affective computing, social signal processing, and behavioral science. This article reports key advances in behavioral science that are becoming possible through these developments. Before beginning, automated facial image analysis and synthesis (AFAS) is briefly described.

Non-ideal class non-point light source quotient image for face relighting

Quotient Image (QI) algorithm has been widely used in face recognition and re-rendering under varying illumination conditions. One of the inaccuracies of QI algorithm is the assumption of “Ideal Class”, that all faces have the same surface normal (3D shape). However, in practice this assumption is often not true. To reduce the inaccuracy, the Non-Ideal Class Non-Point Light source QI (NIC-NPL-QI), which ignores the “Ideal Class” assumption, is developed in this paper for face relighting. Unlike that in the basic QI algorithm a fixed reference object for all test objects is used, in the NIC-NPL-QI algorithm a special reference object for each test object is constructed, so that the test and reference objects have similar illumination images, achieving the equal effect of “Ideal Class” assumption. In the proposed method, the wavelet algorithm is introduced to estimate an illumination image. Furthermore, the proposed NIC-NPL-QI algorithm can handle the harmonic light and shadows. Experiments on Extended Yale B and CMU-PIE databases show that NIC-NLP-QI algorithm obtains better quality in synthesizing face images as compared with state-of-the-art algorithms.

An Optical Flow-Based Approach to Robust Face Recognition Under Expression Variations

Face recognition is one of the most intensively studied topics in computer vision and pattern recognition, but few are focused on how to robustly recognize faces with expressions under the restriction of one single training sample per class. A constrained optical flow algorithm, which combines the advantages of the unambiguous correspondence of feature point labeling and the flexible representation of optical flow computation, has been developed for face recognition from expressional face images. In this paper, we propose an integrated face recognition system that is robust against facial expressions by combining information from the computed intraperson optical flow and the synthesized face image in a probabilistic framework. Our experimental results show that the proposed system improves the accuracy of face recognition from expressional face images.

Synthesized views can improve face recognition

AbstractIt is well known that a new face studied from one view is often difficult to identify from another. This viewpoint dependence has detrimental implications for forensic practice. To compensate for this problem, we employed synthesized face images in the training session of a standard old/new recognition task. Observers in the experimental conditions learned one or more synthesized face images along with an original photograph of the face in a different view, whereas observers in the control conditions learned only the original photograph of the face. It was found that the experimental conditions consistently produced better recognition accuracy than the control conditions. We conclude that synthesized face views can be used to facilitate person identification in forensic applications. Copyright © 2008 John Wiley &amp; Sons, Ltd.