Facial Animation Synthesis Research Articles

Disentangling audio content and emotion with adaptive instance normalization for expressive facial animation synthesis

Abstract3D facial animation synthesis from audio has been a focus in recent years. However, most existing literature works are designed to map audio and visual content, providing limited knowledge regarding the relationship between emotion in audio and expressive facial animation. This work generates audio‐matching facial animations with the specified emotion label. In such a task, we argue that separating the content from audio is indispensable—the proposed model must learn to generate facial content from audio content while expressions from the specified emotion. We achieve it by an adaptive instance normalization module that isolates the content in the audio and combines the emotion embedding from the specified label. The joint content‐emotion embedding is then used to generate 3D facial vertices and texture maps. We compare our method with state‐of‐the‐art baselines, including the facial segmentation‐based and voice conversion‐based disentanglement approaches. We also conduct a user study to evaluate the performance of emotion conditioning. The results indicate that our proposed method outperforms the baselines in animation quality and expression categorization accuracy.

FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis

Talking face synthesis has been widely studied in either appearance-based or warping-based methods. Previous works mostly utilize single face image as a source, and generate novel facial animations by merging other person's facial features. However, some facial regions like eyes or teeth, which may be hidden in the source image, can not be synthesized faithfully and stably. In this paper, We present a landmark driven two-stream network to generate faithful talking facial animation, in which more facial details are created, preserved and transferred from multiple source images instead of a single one. Specifically, we propose a network consisting of a learning and fetching stream. The fetching sub-net directly learns to attentively warp and merge facial regions from five source images of distinctive landmarks, while the learning pipeline renders facial organs from the training face space to compensate. Compared to baseline algorithms, extensive experiments demonstrate that the proposed method achieves a higher performance both quantitatively and qualitatively. Codes are at https://github.com/kgu3/FLNet_AAAI2020.

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

Generative Adversarial Networks (GANs) have achieved impressive results in various image synthesis tasks, and are becoming a hot topic in computer vision research because of the impressive performance they achieved in various applications. In this paper, we introduce the recent research on GANs in the field of image processing, including image synthesis, image generation, image semantic editing, image-to-image translation, image super-resolution, image inpainting, and cartoon generation. We analyze and summarize the methods used in these applications which have improved the generated results. Then, we discuss the challenges faced by GANs and introduce some methods to deal with these problems. We also preview some likely future research directions in the field of GANs, such as video generation, facial animation synthesis and 3D face reconstruction. The purpose of this review is to provide insights into the research on GANs and to present the various applications based on GANs in different scenarios.

Research Paper on Expressive Facial Animation Synthesis by Learning Speech Co-articulation and Expression Spaces

Research Paper on Expressive Facial Animation Synthesis by Learning Speech Co-articulation and Expression Spaces

Realistic emotion visualization by combining facial animation and hairstyle synthesis

Facial expressions are one of most intuitive way for expressing emotions, and can facilitate human-computer interaction by enabling users to communicate with computers using more natural ways. Besides, the hair can be designed to enhance the expression of emotions particularly. To visualize the emotions in multiple aspects for completeness, we propose a realistic visual emotional synthesis system based on the combination of facial expression and hairstyle in this paper. Firstly, facial expression is synthesized by the anatomical model and parameterized model. Secondly, cartoonish hairstyle is synthesized to describe emotion implicitly by the mass-spring model and cantilever beam model. Finally, the synthesis results of facial expression and hairstyle are combined to produce a complete visual emotion synthesis result. Experiment results demonstrate the proposed system can synthesize realistic animation, and the emotion expressiveness by combining of face and hair outperform that by face or hair alone.

Speech driven photo realistic facial animation based on an articulatory DBN model and AAM features

This paper presents a photo realistic facial animation synthesis approach based on an audio visual articulatory dynamic Bayesian network model (AF_AVDBN), in which the maximum asynchronies between the articulatory features, such as lips, tongue and glottis/velum, can be controlled. Perceptual Linear Prediction (PLP) features from audio speech, as well as active appearance model (AAM) features from face images of an audio visual continuous speech database, are adopted to train the AF_AVDBN model parameters. Based on the trained model, given an input audio speech, the optimal AAM visual features are estimated via a maximum likelihood estimation (MLE) criterion, which are then used to construct face images for the animation. In our experiments, facial animations are synthesized for 20 continuous audio speech sentences, using the proposed AF_AVDBN model, as well as the state-of-art methods, being the audio visual state synchronous DBN model (SS_DBN) implementing a multi-stream Hidden Markov Model, and the state asynchronous DBN model (SA_DBN). Objective evaluations on the learned AAM features show that much more accurate visual features can be learned from the AF_AVDBN model. Subjective evaluations show that the synthesized facial animations using AF_AVDBN are better than those using the state based SA_DBN and SS_DBN models, in the overall naturalness and matching accuracy of the mouth movements to the speech content.

Position‐based facial animation synthesis

ABSTRACTWe propose an integrated facial dynamics model addressing the animation of 3D humanoid faces in real time. The computational model mimics facial motion by reproducing the layered anatomical structure of a human head including the bony structure, overlapping facial muscles and the skin. The model is flexible enough to animate face meshes of various shape, connectivity, and scale. Different from previously proposed approaches based on mass–spring networks, overshooting problems are avoided by simulating the dynamics through a position‐based scheme, which allows for real‐time performance, control, and robustness. Experiments demonstrate that convincing expressive facial animation can be interactively prototyped on consumer class platforms. Copyright © 2012 John Wiley & Sons, Ltd.

Evaluation of an image-based talking head with realistic facial expression and head motion

In this paper, we present an image-based talking head system that is able to synthesize flexible head motion and realistic facial expression accompanying speech, given arbitrary text input and control tags. The goal of facial animation synthesis is to generate lip synchronized and natural animations. The talking head is evaluated objectively and subjectively.

Parametric Facial Expression Synthesis and Animation

In this paper, we proposed a parametric model based facial animation synthesis. In the proposed algorithm various facial expressions are generated using the parametric models and the novelty of the algorithm is that new expressions can be generated by blending selected expressions. A complete parameterized model for faces would allow the animator to create any facial image by specifying the appropriate set of parameter values. Using the interface developed along with the proposed algorithm, these images can be used to develop an animation based on an interactive system, virtual reality and games. Interface allows selection of linear and non linear interpolation, play animation and save animation. In this paper we have highlighted advantages and disadvantages of linear and non linear interpolation and proposed a method which incorporates advantages of both. General Terms Animation, Computer graphics.

Expressive Speech Animation Synthesis with Phoneme‐Level Controls

Abstract This paper presents a novel data‐driven expressive speech animation synthesis system with phoneme‐level controls. This system is based on a pre‐recorded facial motion capture database, where an actress was directed to recite a pre‐designed corpus with four facial expressions (neutral, happiness, anger and sadness). Given new phoneme‐aligned expressive speech and its emotion modifiers as inputs, a constrained dynamic programming algorithm is used to search for best‐matched captured motion clips from the processed facial motion database by minimizing a cost function. Users optionally specify ‘hard constraints’ (motion‐node constraints for expressing phoneme utterances) and ‘soft constraints’ (emotion modifiers) to guide this search process. We also introduce a phoneme–Isomap interface for visualizing and interacting phoneme clusters that are typically composed of thousands of facial motion capture frames. On top of this novel visualization interface, users can conveniently remove contaminated motion subsequences from a large facial motion dataset. Facial animation synthesis experiments and objective comparisons between synthesized facial motion and captured motion showed that this system is effective for producing realistic expressive speech animations.

Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces

Synthesizing expressive facial animation is a very challenging topic within the graphics community. In this paper, we present an expressive facial animation synthesis system enabled by automated learning from facial motion capture data. Accurate 3D motions of the markers on the face of a human subject are captured while he/she recites a predesigned corpus, with specific spoken and visual expressions. We present a novel motion capture mining technique that "learns" speech coarticulation models for diphones and triphones from the recorded data. A Phoneme-Independent Expression Eigenspace (PIEES) that encloses the dynamic expression signals is constructed by motion signal processing (phoneme-based time-warping and subtraction) and Principal Component Analysis (PCA) reduction. New expressive facial animations are synthesized as follows: First, the learned coarticulation models are concatenated to synthesize neutral visual speech according to novel speech input, then a texture-synthesis-based approach is used to generate a novel dynamic expression signal from the PIEES model, and finally the synthesized expression signal is blended with the synthesized neutral visual speech to create the final expressive facial animation. Our experiments demonstrate that the system can effectively synthesize realistic expressive facial animation.

Emotional face expression profiles supported by virtual human ontology

Expressive facial animation synthesis of human like characters has had many approaches with good results. MPEG-4 standard has functioned as the basis of many of those approaches. In this paper we w...

Emotional face expression profiles supported by virtual human ontology

AbstractExpressive facial animation synthesis of human like characters has had many approaches with good results. MPEG‐4 standard has functioned as the basis of many of those approaches. In this paper we would like to lay out the knowledge of some of those approaches inside an ontology in order to support the modeling of emotional facial animation in virtual humans (VH). Inside this ontology we will present MPEG‐4 facial animation concepts and its relationship with emotion through expression profiles that utilize psychological models of emotions. The ontology allows storing, indexing and retrieving prerecorded synthetic facial animations that can express a given emotion. Also this ontology can be used a refined knowledge base in regards to the emotional facial animation creation. This ontology is made using Web Ontology Language and the results are presented as answered queries. Copyright © 2006 John Wiley & Sons, Ltd.

Surface detail capturing for realistic facial animation

In this paper, a facial animation system is proposed for capturing both geometrical information and illumination changes of surface details, called expression details, from video clips simultaneously, and the captured data can be widely applied to different 2D face images and 3D face models. While tracking the geometric data, we record the expression details by ratio images. For 2D facial animation synthesis, these ratio images are used to generate dynamic textures. Because a ratio image is obtained via dividing colors of an expressive face by those of a neutral face, pixels with ratio value smaller than one are where a wrinkle or crease appears. Therefore, the gradients of the ratio value at each pixel in ratio images are regarded as changes of a face surface, and original normals on the surface can be adjusted according to these gradients. Based on this idea, we can convert the ratio images into a sequence of normal maps and then apply them to animated 3D model rendering. With the expression detail mapping, the resulted facial animations are more life-like and more expressive.