Facial Animation System Research Articles

A statistical shape modeling approach for predicting subject-specific human skull from head surface.

Human skull is an important body structure for jaw movement and facial mimic simulations. Surface head can be reconstructed using 3D scanners in a straightforward way. However, internal skull is challenging to be generated when only external information is available. Very few studies in the literature focused on the skull generation from outside head information, especially in a subject-specific manner with a complete skull. Consequently, this present study proposes a novel process for predicting a subject-specific skull with full details from a given head surface using a statistical shape modeling approach. Partial least squared regression (PLSR)-based method was used. A CT image database of 209 subjects (genders-160 males and 49 females; ages-34-88years) was used for learning head-to-skull relationship. Heads and skulls were reconstructed from CT images to extract head/skull feature points, head/skull feature distances, head-skull thickness, and head/skull volume descriptors for the learning process. A hyperparameter turning process was performed to determine the optimal numbers of head/skull feature points, PLSR components, deformation control points, and appropriate learning strategies for our learning problem. Two learning strategies (point-to-thickness with/without volume descriptor and distance-to-thickness with/without volume descriptor) were proposed. Moreover, a 10-fold cross-validation procedure was conducted to evaluate the accuracy of the proposed learning strategies. Finally, the best and worst reconstructed skulls were analyzed based on the best learning strategy with its optimal parameters. The optimal number of head/skull feature points and deformation control points are 2300 and 1300 points, respectively. The optimal number of PLSR components ranges from 4 to 8 for all learning configurations. Cross-validation showed that grand means and standard deviations of the point-to-thickness, point-to-thickness with volumes, distance-to-thickness, and distance-to-thickness with volumes learning configurations are 2.48 ± 0.27mm, 2.46 ± 0.19mm, 2.46 ± 0.15mm, and 2.48 ± 0.22mm, respectively. Thus, the distance-to-thickness is the best learning configuration for our head-to-skull prediction problem. Moreover, the mean Hausdorff distances are 2.09 ± 0.15mm and 2.64 ± 0.26mm for the best and worst predicted skull, respectively. A novel head-to-skull prediction process based on the PLSR method was developed and evaluated. This process allows, for the first time, predicting 3D subject-specific human skulls from head surface information with a very good accuracy level. As perspective, the proposed head-to-skull prediction process will be integrated into our real-time computer-aided vision system for facial animation and rehabilitation. Graphical abstract.

How Actors Can Animate Game Characters: Integrating Performance Theory in the Emotion Model of a Game Character

Despite the development of sophisticated emotion models, game character facial animation is still often completed with laborious hand-controlled key framing, only marginally assisted by automation. Behavior trees and animation state machines are used mostly to manage animation transitions of physical business, like walking or lifting objects. Attempts at automated facial animation rely on discrete facial iconic emotion poses, thus resulting in mechanical “acting”. The techniques of acting instructor and theorist Sanford Meisner reveal a process of role development whose character model resembles components of Appraisal Theory. The similarity conjures an experiment to discover if an “emotion engine” and workflow method can model the emotions of an autonomous animated character using actor-centric techniques. Success would allow an animation director to autonomously animate a character’s face with the subtlety of gradient expressions. Using a head-shot video stream of one of two actors performing a structured Meisner-esque improvisation as the primary data source, this research demonstrates the viability of an actor-centric workflow to create an autonomous facial animation system.

Real-Time 3-D Facial Animation: From Appearance to Internal Articulators

A real-time 3-D facial animation system produces animation for appearance and internal articulators. For appearance, an anatomical model, including the skeleton, muscle, and skin, is built based on anatomical characteristics and a data-driven model is obtained by learning the mapping between texture and depth. Then, the two models are combined to produce animations with various strengths, since the anatomical model can control the animation strength directly and the data-driven model can capture the nuances of facial motion. For internal articulators, tongue tissue arrangements are obtained from medical data. Then, a nonlinear, quasi-incompressible, isotropic, hyperelastic biomechanical model is applied to describe tongue tissues and an anisotropic biomechanical model is applied to reflect the active and passive mechanical behavior of tongue muscle fibers. The tongue animation is simulated using the finite-element method for realism, while the collisions between the tongue and other articulators are simulated with a mass-spring model for efficiency. Experiments show that the system achieves high perceptual evaluation scores and quantitative improvements are demonstrated in the objective evaluation and user studies, compared with the outputs of other systems.

An Extensible Game Engine to Develop Animated Facial Avatars in 3D Virtual Environment

Avatar facial expression and animation in 3D Collaborative Virtual Environment (CVE) systems are reconstructed through a complex manipulation of all details that compose it like muscles, bones and wrinkles in 3D space. The need for a fast and easy reconstruction approach has emerged out in the recent years due to its application in various domains; 3D disaster management and military training etc. These details simulation must be as realistic as possible to convey different emotions according to the constantly changing situations in CVE during the runtime. For example, in 3D disaster management, it is important to use dynamic avatar emotions; firefighters should be frightened when dealing with a fire disaster and smiling when treating injures and evacuating habitants. However, the solution of facial animation remains a challenge that restricts the rapid and ease development of facial animation systems. In this work, the author presents extensible game engine architecture to easily produce real-time facial animations using a script atomic action without having to deal with control structures and 3D programing language. The proposed architecture defines various controllers, object behaviors, tactical and graphics rendering, and collision effects to quickly design 3D virtual environment. Firstly, the author gives the concept of atomic expression, and the method to build a parametrized script file according to the atomic expression. Then the author shows the validity of the generated expressions based on the MPEG-4 facial animation framework. Finally, the feasibility of the proposed architecture is tested via a firefighter scenario. The author's approach has the advantages over previous techniques of fitting directly an easy and faster technology with a high degree of programming independence. The author also minimizes the interaction with the game engine during the runtime by injecting dynamically the XML file into the game engine without stopping or restarting the engine.

Blend Shape Interpolation and FACS for Realistic Avatar

The quest of developing realistic facial animation is ever-growing. The emergence of sophisticated algorithms, new graphical user interfaces, laser scans and advanced 3D tools imparted further impetus towards the rapid advancement of complex virtual human facial model. Face-to-face communication being the most natural way of human interaction, the facial animation systems became more attractive in the information technology era for sundry applications. The production of computer-animated movies using synthetic actors are still challenging issues. Proposed facial expression carries the signature of happiness, sadness, angry or cheerful, etc. The mood of a particular person in the midst of a large group can immediately be identified via very subtle changes in facial expressions. Facial expressions being very complex as well as important nonverbal communication channel are tricky to synthesize realistically using computer graphics. Computer synthesis of practical facial expressions must deal with the geometric representation of the human face and the control of the facial animation. We developed a new approach by integrating blend shape interpolation (BSI) and facial action coding system (FACS) to create a realistic and expressive computer facial animation design. The BSI is used to generate the natural face while the FACS is employed to reflect the exact facial muscle movements for four basic natural emotional expressions such as angry, happy, sad and fear with high fidelity. The results in perceiving the realistic facial expression for virtual human emotions based on facial skin color and texture may contribute towards the development of virtual reality and game environment of computer aided graphics animation systems. Realistic facial expressions of avatar.

Synthesizing Performance-driven Facial Animation

In this paper, we present a system for real-time performance-driven facial animation. With the system, the user can control the facial expression of a digital character by acting out the desired facial action in front of an ordinary camera. First, we create a muscle-based 3D face model. The muscle actuation parameters are used to animate the face model. To increase the reality of facial animation, the orbicularis oris in our face model is divided into the inner part and outer part. We also establish the relationship between jaw rotation and facial surface deformation. Second, a real-time facial tracking method is employed to track the facial features of a performer in the video. Finally, the tracked facial feature points are used to estimate muscle actuation parameters to drive the face model. Experimental results show that our system runs in real time and outputs realistic facial animations. Compared with most existing performance-based facial animation systems, ours does not require facial markers, intrusive lighting, or special scanning equipment, thus it is inexpensive and easy to use.

A video, text, and speech-driven realistic 3-d virtual head for human-machine interface.

A multiple inputs-driven realistic facial animation system based on 3-D virtual head for human-machine interface is proposed. The system can be driven independently by video, text, and speech, thus can interact with humans through diverse interfaces. The combination of parameterized model and muscular model is used to obtain a tradeoff between computational efficiency and high realism of 3-D facial animation. The online appearance model is used to track 3-D facial motion from video in the framework of particle filtering, and multiple measurements, i.e., pixel color value of input image and Gabor wavelet coefficient of illumination ratio image, are infused to reduce the influence of lighting and person dependence for the construction of online appearance model. The tri-phone model is used to reduce the computational consumption of visual co-articulation in speech synchronized viseme synthesis without sacrificing any performance. The objective and subjective experiments show that the system is suitable for human-machine interaction.

A 3D face animation system for mobile devices

In this paper, we present a 3D face animation system rendered on mobile devices. The system automatically creates realistic facial animation from text input with emotion tags. First, an input string is converted into synthetic voice and phonetic information. Then, 3D head model performs facial movements synchronized to the speech. The proposed system offers an affordable quick solution for applications that require virtual actors speaking text in which human-machine interfaces on mobile devices can profit.

A New Method of 3D Facial Expression Animation

Animating expressive facial animation is a very challenging topic within the graphics community. In this paper, we introduce a novel ERI (expression ratio image) driving framework based on SVR and MPEG-4 for automatic 3D facial expression animation. Through using the method of support vector regression (SVR), the framework can learn and forecast the regression relationship between the facial animation parameters (FAPs) and the parameters of expression ratio image. Firstly, we build a 3D face animation system driven by FAP. Secondly, through using the method of principle component analysis (PCA), we generate the parameter sets of eigen-ERI space, which will rebuild reasonable expression ratio image. Then we learn a model with the support vector regression mapping, and facial animation parameters can be synthesized quickly with the parameters of eigen-ERI. Finally, we implement our 3D face animation system driving by the result of FAP and it works effectively.

Real-time facial animation on mobile devices

We present a performance-based facial animation system capable of running on mobile devices at real-time frame rates. A key component of our system is a novel regression algorithm that accurately infers the facial motion parameters from 2D video frames of an ordinary web camera. Compared with the state-of-the-art facial shape regression algorithm [1], which takes a two-step procedure to track facial animations (i.e., first regressing the 3D positions of facial landmarks, and then computing the head poses and expression coefficients), we directly regress the head poses and expression coefficients. This one-step approach greatly reduces the dimension of the regression target and significantly improves the tracking performance while preserving the tracking accuracy. We further propose to collect the training images of the user under different lighting environments, and make use of the data to learn a user-specific regressor, which can robustly handle lighting changes that frequently occur when using mobile devices.

3D shape regression for real-time facial animation

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated.

A comprehensive system for facial animation of generic 3D head models driven by speech

A comprehensive system for facial animation of generic 3D head models driven by speech is presented in this article. In the training stage, audio-visual information is extracted from audio-visual training data, and then used to compute the parameters of a single joint audio-visual hidden Markov model (AV-HMM). In contrast to most of the methods in the literature, the proposed approach does not require segmentation/classification processing stages of the audio-visual data, avoiding the error propagation related to these procedures. The trained AV-HMM provides a compact representation of the audio-visual data, without the need of phoneme (word) segmentation, which makes it adaptable to different languages. Visual features are estimated from the speech signal based on the inversion of the AV-HMM. The estimated visual speech features are used to animate a simple face model. The animation of a more complex head model is then obtained by automatically mapping the deformation of the simple model to it, using a small number of control points for the interpolation. The proposed algorithm allows the animation of 3D head models of arbitrary complexity through a simple setup procedure. The resulting animation is evaluated in terms of intelligibility of visual speech through perceptual tests, showing a promising performance. The computational complexity of the proposed system is analyzed, showing the feasibility of its real-time implementation.

Three-dimensional cartoon facial animation based on art rules

Three-dimensional (3D) cartoon facial animation is one step further than the challenging 3D caricaturing which generates 3D still caricatures only. In this paper, a 3D cartoon facial animation system is developed for a subject given only a single frontal face image of a neutral expression. The system is composed of three steps consisting of 3D cartoon face exaggeration, texture processing, and 3D cartoon facial animation. By following caricaturing rules of artists, instead of mathematical formulations, 3D cartoon face exaggeration is accomplished at both global and local levels. As a result, the final exaggeration is capable of depicting the characteristics of an input face while achieving artistic deformations. In the texture processing step, texture coordinates of the vertices of the cartoon face model are obtained by mapping the parameterized grid of the standard face model to a cartoon face template and aligning the input face to the face template. Finally, 3D cartoon facial animation is implemented in the MPEG-4 animation framework. In order to avoid time-consuming construction of a face animation table, we propose to utilize the tables of existing models through model mapping. Experimental results demonstrate the effectiveness and efficiency of our proposed system.

Design and Implementation of 3D Facial Animation Based on MPEG-4

Through the in-depth study of the MPEG-4 face model definition standard and animation-driven principles, learning from the existing generation technology of facial animation, this paper presents a 3D facial animation system design program. This program can accept driver information to generate a realistic facial expression animation and simulate the real face actions. At the same time, in the implementation process it also uses FAP frame with a mask and implementation method of FAP intermediate frame calculation, insert to reduce the amount of animation-driven data, and then improve the continuous effect of facial animation.

Real-time speech-driven animation of expressive talking faces

In this paper, we present a real-time facial animation system in which speech drives mouth movements and facial expressions synchronously. Considering five basic emotions, a hierarchical structure with an upper layer of emotion classification is established. Based on the recognized emotion label, the under-layer classification at sub-phonemic level has been modelled on the relationship between acoustic features of frames and audio labels in phonemes. Using certain constraint, the predicted emotion labels of speech are adjusted to gain the facial expression labels which are combined with sub-phonemic labels. The combinations are mapped into facial action units (FAUs), and audio-visual synchronized animation with mouth movements and facial expressions is generated by morphing between FAUs. The experimental results demonstrate that the two-layer structure succeeds in both emotion and sub-phonemic classifications, and the synthesized facial sequences reach a comparative convincing quality.

Method and System for Aligning Natural and Synthetic Video to Speech Synthesis

According to MPEG-4's TTS architecture, facial animation can be driven by two streams simultaneously—text, and Facial Animation Parameters. In this architecture, text input is sent to a Text-To-Speech converter at a decoder that drives the mouth shapes of the face. Facial Animation Parameters are sent from an encoder to the face over the communication channel. The present invention includes codes (known as bookmarks) in the text string transmitted to the Text-to-Speech converter, which bookmarks are placed between words as well as inside them. According to the present invention, the bookmarks carry an encoder time stamp. Due to the nature of text-to-speech conversion, the encoder time stamp does not relate to real-world time, and should be interpreted as a counter. In addition, the Facial Animation Parameter stream carries the same encoder time stamp found in the bookmark of the text. The system of the present invention reads the bookmark and provides the encoder time stamp as well as a real-time time stamp to the facial animation system. Finally, the facial animation system associates the correct facial animation parameter with the real-time time stamp using the encoder time stamp of the bookmark as a reference.

A Facial Animation System Based on Single Image

A Facial Animation System Based on Single Image

The Digital Emily Project: Achieving a Photorealistic Digital Actor

The Digital Emily Project uses advanced face scanning, character rigging, performance capture, and compositing to achieve one of the world's first photorealistic digital facial performances. The project scanned the geometry and reflectance of actress Emily O'Brien's face in 33 poses, showing different emotions, gaze directions, and lip formations in a light stage. These high-resolution scans-accurate to skin pores and fine wrinkles-became the basis for building a blendshape-based facial-animation rig whose expressions closely matched the scans. The blendshape rig drove displacement maps to add dynamic surface detail. A video-based facial animation system animated the face according to the performance in a reference video, and the digital face was tracked onto the video's motion and rendered under the same illumination. The result was a realistic 3D digital facial performance credited as one of the first to cross the "uncanny valley" between animated and fully human performances.

Generating anatomical substructures for physically-based facial animation. Part 1: A methodology for skull fitting

Physically-based facial animation techniques are capable of producing realistic facial deformations, but have failed to find meaningful use outside the academic community because they possess the disadvantages of being much harder to create, and even harder to reuse in comparison to other methods of facial animation. As a first step toward a physically-based facial animation system that is truly reusable, this paper outlines a landmark-based process for fitting a generic skull to any given face model, using thin-plate splines and extended kriging predictor deformers, and an interactive scaling technique for incorporating experimentally obtained soft-tissue depth data into the morphing process.

Facial performance synthesis using deformation-driven polynomial displacement maps

We present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed facial geometry is inferred, and a synthesis phase where novel detailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displacements between a neutral mesh driven by the motion capture markers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dynamic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.