Facial Animation Parameters Research Articles

A Talking Face Driven by Voice using Hidden Markov Model

In this paper, we utilized Hidden Markov Model (HMM) as a mapping mechanism between two different kinds of correlated signals. Specifically, we developed a voice-driven talking head system by exploiting the physical relationships between the shape of the mouth and the sound that is produced. The proposed system can be easily trained and a talking head can be efficiently animated. In the training phase, the Mel-scale Frequency Cepstral Coefficients (MFCC) were analyzed from audio signals and the Facial Animation Parameters (FAP) were extracted from video signals. Then both audio and video features were integrated to train a single HMM. In the synthesis phase, the HMM was used to correlate a completely novel audio track to a FAP sequence for face synthesis with the help of Facial Animation Engine (FAE). The experiments demonstrated the effects of the proposed voice-driven talking head on both man and woman, with two kinds of styles (speaking and singing) and using three kinds of languages (Chinese, English and Taiwanese). The possible applications of the proposed system are computer aided instruction, online guide, virtual conference, lip synchronization, human computer interaction and so on.

Automatic Facial Expression Recognition Using Facial Animation Parameters and Multistream HMMs

The performance of an automatic facial expression recognition system can be significantly improved by modeling the reliability of different streams of facial expression information utilizing multistream hidden Markov models (HMMs). In this paper, we present an automatic multistream HMM facial expression recognition system and analyze its performance. The proposed system utilizes facial animation parameters (FAPs), supported by the MPEG-4 standard, as features for facial expression classification. Specifically, the FAPs describing the movement of the outer-lip contours and eyebrows are used as observations. Experiments are first performed employing single-stream HMMs under several different scenarios, utilizing outer-lip and eyebrow FAPs individually and jointly. A multistream HMM approach is proposed for introducing facial expression and FAP group dependent stream reliability weights. The stream weights are determined based on the facial expression recognition results obtained when FAP streams are utilized individually. The proposed multistream HMM facial expression system, which utilizes stream reliability weights, achieves relative reduction of the facial expression recognition error of 44% compared to the single-stream HMM system.

Parametric Generation of Facial Expressions Based on FACS

Abstract This paper presents a parametric performance‐based model for facial animation that was inspired by Facial Action Coding System (FACS) developed by P. Ekman and F. W. Friesen. The FACS consists of 44 Action Units (AUs) corresponding to visual changes on the face. Additionally, predefined co‐occurrence rules describe how different AUs influence each other. In our model, each facial animation parameter corresponds to one of the AUs as defined in FACS. Implementation of the model is completed with methods for accumulating displacement from separate AUs together, and fuzzy‐logical adaptation of co‐occurrence rules from the FACS. We also describe the method for adapting our model to a specific person.

Emotion recognition through facial expression analysis based on a neurofuzzy network

Extracting and validating emotional cues through analysis of users' facial expressions is of high importance for improving the level of interaction in man machine communication systems. Extraction of appropriate facial features and consequent recognition of the user's emotional state that can be robust to facial expression variations among different users is the topic of this paper. Facial animation parameters (FAPs) defined according to the ISO MPEG-4 standard are extracted by a robust facial analysis system, accompanied by appropriate confidence measures of the estimation accuracy. A novel neurofuzzy system is then created, based on rules that have been defined through analysis of FAP variations both at the discrete emotional space, as well as in the 2D continuous activation–evaluation one. The neurofuzzy system allows for further learning and adaptation to specific users' facial expression characteristics, measured though FAP estimation in real life application of the system, using analysis by clustering of the obtained FAP values. Experimental studies with emotionally expressive datasets, generated in the EC IST ERMIS project indicate the good performance and potential of the developed technologies.

Internet communication using real-time facial expression analysis and synthesis

In this paper, the authors have developed a system that animates 3D facial agents based on real-time facial expression analysis techniques and research on synthesizing facial expressions and text-to-speech capabilities. This system combines visual, auditory, and primary interfaces to communicate one coherent multimodal chat experience. Users can represent themselves using agents they select from a group that we have predefined. When a user shows a particular expression while typing a text, the 3D agent at the receiving end speaks the message aloud while it replays the recognized facial expression sequences and also augments the synthesized voice with appropriate emotional content. Because the visual data exchange is based on the MPEG-4 high-level Facial Animation Parameter for facial expressions (FAP 2), rather than real-time video, the method requires very low bandwidth.

Speech-To-Video Synthesis Using MPEG-4 Compliant Visual Features

There is a strong correlation between the building blocks of speech (phonemes) and the building blocks of visual speech (visimes). In this paper, this correlation is exploited and an approach is proposed for synthesizing the visual representation of speech from a narrow-band acoustic speech signal. The visual speech is represented in terms of the facial animation parameters (FAPs), supported by the MPEG-4 standard. The main contribution of this paper is the development of a correlation hidden Markov model (CHMM) system, which integrates independently trained acoustic HMM (AHMM) and visual HMM (VHMM) systems, in order to realize speech-to-video synthesis. The proposed CHMM system allows for different model topologies for acoustic and visual HMMs. It performs late integration and reduces the amount of required training data compared to early integration modeling techniques. Temporal accuracy experiments, comparison of the synthesized FAPs to the original FAPs, and audio-visual automatic speech recognition (AV-ASR) experiments utilizing the synthesized visual speech were performed in order to objectively measure the performance of the system. The objective experiments demonstrated that the proposed approach reduces time alignment errors by 40.5% compared to the conventional temporal scaling method, that the synthesized FAP sequences are very similar to the original FAP sequences, and that synthesized FAP sequences contain visual speechreading information that can improve AV-ASR performance.

Optimizing facial animation parameters for MPEG-4

This paper presents a method for automatic optimization of FAPs (facial animation parameters) based on a face video sequence. An FAP optimizer unit compares a rendered face with the original face in the video sequence. The difference is iteratively minimized to produce an optimal set of FAPs. Unnecessary FAPs are eliminated and the search is partitioned to speed up the optimization process. Three different search techniques, steepest descent method, linear search method, and cyclic coordinates method are applied to derive an optimum. The FAPs are very accurate and the animation based on the FAPs is lifelike and truthful to the original video.

Learning and synthesizing MPEG-4 compatible 3-D face animation from video sequence

We present a new system that applies an example-based learning method to learn facial motion patterns from a video sequence of individual facial behavior such as lip motion and facial expressions, and using that to create vivid three-dimensional (3-D) face animation according to the definition of MPEG-4 face animation parameters. The system consists of three key modules, face tracking, pattern learning, and face animation. In face tracking, to reduce the complexity of the tracking process, a novel coarse-to-fine strategy combined with a Kalman filter is proposed for localizing key facial landmarks in each image of the video. The landmarks' sequence is normalized into a visual feature matrix and then fed to the next step of process. In pattern learning, in the pretraining stage, the parameters of the camera that took the video are requested with the training video data so the system can estimate the basic mapping from a normalized two-dimensional (2-D) visual feature matrix to the representation in 3-D MPEG-4 face animation parameter space, in assistance with the computer vision method. In the practice stage, considering that in most cases camera parameters are not provided with video data, the system uses machine learning technology to complement the incomplete 3-D information for the mapping that information is needed in face orientation presentation. The example-based learning in this system integrates several methods including clustering, HMM, and ANN to make a better conversion from a 2-D to 3-D model and better estimation of incomplete 3-D information for good mapping; this will be used to drive face animation thereafter. In face animation, the system can synthesize face animation following any type of face motion in video. Experiments show that our system produces more vivid face motion animation, compared to other early systems.

From Greta's mind to her face: modelling the dynamics of affective states in a conversational embodied agent

This paper describes the results of a research project aimed at implementing a ‘realistic’ 3D Embodied Agent that can be animated in real-time and is ‘believable and expressive’: that is, able to coherently communicate complex information through the combination and the tight synchronisation of verbal and nonverbal signals. We describe, in particular, how we ‘animate’ this Agent (that we called Greta) so as to enable her to manifest the affective states that are dynamically activated and de-activated in her mind during the dialog with the user. The system is made up of three tightly interrelated components: • A representation of the Agent Mind: this includes long and short-term affective components (personality and emotions) and simulates how emotions are triggered and decay over time according to the Agent's personality and to the context, and how several emotions may overlap. Dynamic belief networks with weighting of goals is the formalism we employ to this purpose. • A mark-up language to denote the communicative meanings that may be associated with dialog moves performed by the Agent. • A translation of the Agent's tagged move into a face expression, that combines appropriately the available channels (gaze direction, eyebrow shape, head direction and movement etc). The final output is a 3-D facial model that respects the MPEG-4 standard and uses MPEG-4 Facial Animation Parameters to produce facial expressions. Throughout the paper, we illustrate the results obtained, with an example of dialog in the domain of ‘Advice about eating disorders’. The paper concludes with an analysis of advantages of our cognitive model of emotion triggering and of the problems found in testing it. Although we did not yet complete a formal evaluation of our system, we briefly describe how we plan to assess the agent's believability in terms of consistency of its communicative behaviour.

MPEG‐4 facial animation in video analysis and synthesis

AbstractMPEG‐4 supports the definition, encoding, transmission, and animation of 3‐D head and body models. These features can be used for a variety of different applications ranging from low bit‐rate video coding to character and avatar animation. In this article, an entire system for the analysis of facial expressions from image sequences and their synthesis is presented. New methods for the estimation of MPEG‐4 facial animation parameters as well as scene illumination are proposed. Experiments for different applications demonstrate the potential of using facial animation techniques in video analysis and synthesis. A model‐based codec is presented that is able to encode head‐and‐shoulder video sequences at bit‐rates of about 1 kbit/s. Besides the low bit‐rate, many enhancements and scene modifications can be easily applied, like scene lighting changes or cloning of expressions for character animation. But also for the encoding of arbitrary sequences, 3‐D knowledge can help to increase the coding efficiency. With our model‐aided codec, bit‐rate reductions of up to 45% at the same quality can be achieved in comparison to standard hybrid video codecs. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13, 245–256, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10072

Audio-Visual Speech Recognition Using MPEG-4 Compliant Visual Features

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial animation parameters (FAPs) supported by the MPEG-4 standard for the visual representation of speech. We also describe a robust and automatic algorithm we have developed to extract FAPs from visual data, which does not require hand labeling or extensive training procedures. The principal component analysis (PCA) was performed on the FAPs in order to decrease the dimensionality of the visual feature vectors, and the derived projection weights were used as visual features in the audio-visual automatic speech recognition (ASR) experiments. Both single-stream and multistream hidden Markov models (HMMs) were used to model the ASR system, integrate audio and visual information, and perform a relatively large vocabulary (approximately 1000 words) speech recognition experiments. The experiments performed use clean audio data and audio data corrupted by stationary white Gaussian noise at various SNRs. The proposed system reduces the word error rate (WER) by 20% to 23% relatively to audio-only speech recognition WERs, at various SNRs (0–30 dB) with additive white Gaussian noise, and by 19% relatively to audio-only speech recognition WER under clean audio conditions.

Parameterized Facial Expression Synthesis Based on MPEG-4

In the framework of MPEG-4, one can include applications where virtual agents, utilizing both textual and multisensory data, including facial expressions and nonverbal speech help systems become accustomed to the actual feelings of the user. Applications of this technology are expected in educational environments, virtual collaborative workplaces, communities, and interactive entertainment. Facial animation has gained much interest within the MPEG-4 framework; with implementation details being an open research area (Tekalp, 1999). In this paper, we describe a method for enriching human computer interaction, focusing on analysis and synthesis of primary and intermediate facial expressions (Ekman and Friesen (1978)). To achieve this goal, we utilize facial animation parameters (FAPs) to model primary expressions and describe a rule-based technique for handling intermediate ones. A relation between FAPs and the activation parameter proposed in classical psychological studies is established, leading to parameterized facial expression analysis and synthesis notions, compatible with the MPEG-4 standard.

A modular approach to facial feature segmentation on real sequences

In this paper a modular approach of gradual confidence for facial feature extraction over real video frames is presented. The problem is being dealt under general imaging conditions and soft presumptions. The proposed methodology copes with large variations in the appearance of diverse subjects, as well as of the same subject in various instances within real video sequences. Areas of the face that statistically seem to be outstanding form an initial set of regions that are likely to include information about the features of interest. Enhancement of these regions produces closed objects, which reveal—through the use of a fuzzy system—a dominant angle, i.e. the facial rotation angle . The object set is restricted using the dominant angle. An exhaustive search is performed among all candidate objects, matching a pattern that models the relative position of the eyes and the mouth. Labeling of the winner features can be used to evaluate the features extracted and provide feedback in an iterative framework. A subset of the MPEG-4 facial definition or facial animation parameter set can be obtained. This gradual feature revelation is performed under optimization for each step, producing a posteriori knowledge about the face and leading to a step-by-step visualization of the features in search.

FAP extraction using three-dimensional motion estimation

An integral part of the MPEG-4 standard is the definition of face animation parameters (FAPs). This paper presents a method for the determination of FAPs by using three dimensional (3-D) rigid and nonrigid motion of human facial features found from two-dimensional (2-D) image sequences. The proposed method assumes that a 3-D model has been fitted to the first frame of the sequence, tracks the motion of characteristic facial features, calculates the 3-D rigid and nonrigid motion of facial features, and through this, estimates the FAPs as defined by the MPEG-4 coding standard. The 2-D tracking process is based on a novel enhanced version of the algorithm proposed by Kanade, Lucas, and Tomasi (1994, 1991). The nonrigid motion estimation is achieved using the same tracking mechanism guided by the facial motion model implied by the MPEG-4 FAPs.

다양한 형식의 얼굴정보와 준원근 카메라 모델해석을 이용한 얼굴 특징점 및 움직임 복원

본 논문은 MPEG4 SNHC의 얼굴 모델 인코딩을 구현하기 위하여 연속된 2차원 영상으로부터 얼굴영역을 검출하고, 얼굴의 특징데이터들을 추출한 후, 얼굴의 3차원 모양 및 움직임 정보를 복원하는 알고리즘과 결과를 제시한다. 얼굴 영역 검출을 위해서 영상의 거리, 피부색상, 움직임 색상정보등을 융합시킨 멀티모달합성의 방법이 사용되었다. 결정된 얼굴영역에서는 MPEG4의 FDP(Face Definition Parameter) 에서 제시된 특징점 위치중 23개의 주요 얼굴 특징점을 추출하며 추출성능을 향상시키기 위하여 GSCD(Generalized Skin Color Distribution), BWCD(Black and White Color Distribution)등의 움직임색상 변환기법과 형태연산 방법이 제시되었다. 추출된 2차원 얼팔 특징점들로부터 얼굴의 3차원 모양, 움직임 정보를 복원하기 위하여 준원근 카메라 모델을 적용하여 SVD(Singular Value Decomposition)에 의한 인수분해연산을 수행하였다. 본 논문에서 제시된 방법들의 성능을 객관적으로 평가하기 위하여 크기와 위치가 알려진 3차원 물체에 대해 실험을 행하였으며, 복원된 얼굴의 움직임 정보는 MPEG4 FAP(Face Animation Parameter)로 변환된 후, 인터넷상에서 확인이 가능한 가상얼굴모델에 인코딩되어 실제 얼굴파 일치하는 모습을 확인하였다. Robust extraction of 3D facial features and global motion information from 2D image sequence for the MPEG-4 SNHC face model encoding is described. The facial regions are detected from image sequence using multi-modal fusion technique that combines range, color and motion information. 23 facial features among the MPEG-4 FDP (Face Definition Parameters) are extracted automatically inside the facial region using color transform (GSCD, BWCD) and morphological processing. The extracted facial features are used to recover the 3D shape and global motion of the object using paraperspective camera model and SVD (Singular Value Decomposition) factorization method. A 3D synthetic object is designed and tested to show the performance of proposed algorithm. The recovered 3D motion information is transformed into global motion parameters of FAP (Face Animation Parameters) of the MPEG-4 to synchronize a generic face model with a real face.

Real-time MPEG-4 facial animation with 3D scalable meshes

In this paper the complete framework of a software package for real-time animation of 3D facial models according to the MPEG-4 specifications is described. The focus is mainly on the algorithmic solution that has been adopted for scaling the polygon mesh to a level of complexity suited to the specific hw/sw profile of the platform that hosts the decoder. The basic functionality of this system is its ability to deform virtual faces in response to MPEG-4 Facial Animation Parameters (FAP) according to “animation rules” whose computation is based on very simple “semantic” information previously associated with the model. The software architecture is modular and each module, implementing a specific functionality, can be easily plugged in or unplugged. In this paper, we begin with a general presentation of the system and then we present a brief description of its main modules: the face model, the mesh animation, the mesh calibration, the mesh simplification and the FAP decoding. While for the other modules a deeper analysis is left to referenced bibliography previously published by the authors, a comprehensive description is given here of the mesh simplification block together with experimental evidence of its efficiency. The algorithm we have adopted for mesh simplification is based on iterative edge contraction with the introduction of specific geometry constraints due to the particular nature of the 3D mesh representing an MPEG-4 animated face. In this paper it will be shown how rather complex 3D facial models, optionally textured, can be simplified in real-time in order to adapt the rendering performance to the hardware capabilities, whilst still maintaining compliance with the MPEG-4 specifications on facial animation.

Visual Estimation and Compression of Facial Motion Parameters—Elements of a 3D Model-Based Video Coding System

The MPEG4 standard supports the transmission and composition of facial animation with natural video by including a facial animation parameter (FAP) set that is defined based on the study of minimal facial actions and is closely related to muscle actions. The FAP set enables model-based representation of natural or synthetic talking head sequences and allows intelligible visual reproduction of facial expressions, emotions, and speech pronunciations at the receiver. This paper describes two key components we have developed for building a model-based video coding system: (1) a method for estimating FAP parameters based on our previously proposed piecewise Bezier volume deformation model (PBVD), and (2) various methods for encoding FAP parameters. PBVD is a linear deformation model suitable for both the synthesis and the analysis of facial images. Each FAP parameter is a basis function in this model. Experimental results on PBVD-based animation, model-based tracking, and spatial-temporal compression of FAP parameters are demonstrated in this paper.

Audiovisual integration in multimedia communications based on MPEG-4 facial animation

Recent progress in audiovisual research shows that joint processing of audio and video provides advantages that are not available when the audio and video are processed independently. The importance of the MPEG-4 standard is in specifying an object-based audiovisual representation framework, integrating both natural and synthetic content. MPEG-4 includes a facial animation parameters (FAP) set that enables model-based representation of natural or synthetic talking-head sequences and allows intelligible visual reproduction of facial expressions, emotions, and speech pronunciations at the receiver. Coding methods for bit-rate reduction of FAPs, which make possible the transmission of multiple talking heads over band-limited channels, are reviewed in this paper. Further, we emphasis relationships between natural/synthetic audiovideo coding from the point of view integration of face animation with natural video. Within MPEG-4, a binary format for scene (BIFS) description framework offers a parametric methodology for scene structure representation and efficient coding for transmission or storage. Also, we address the MPEG-4 profiling strategy in facial animation, which guarantees that the standard can provide adequate solutions for applications in multimedia communications.

A new efficient expression generation and automatic cloning method for multimedia actors

This paper describes a new and efficient method for facial expression generation on cloned synthetic head models. The system uses abstract facial muscles called action units (AUs) based on both anatomical muscles and the facial action coding system. The facial expression generation method has real-time performance, is less computationally expensive than physically based models, and has greater anatomical correspondence than rational free-form deformation or spline-based, techniques. Automatic cloning of a real human head is done by adapting a generic facial and head mesh to Cyberware laser scanned data. The conformation of the generic head to the individual data and the fitting of texture onto it are based on a fully automatic feature extraction procedure. Individual facial animation parameters are also automatically estimated during the conformation process. The entire animation system is hierarchical; emotions and visemes (the visual mouth shapes that occur during speech) are defined in terms of the AUs, and higher-level gestures are defined in terms of AUs, emotions, and visemes as well as the temporal relationships between them. The main emphasis of the paper is on the abstract muscle model, along with limited discussion on the automatic cloning process and higher-level animation control aspects.

An efficient use of MPEG-4 FAP interpolation for facial animation at 70 bits/frame

An efficient algorithm is proposed to exploit the facial animation parameter (FAP) interpolation modality specified by the MPEG-4 standard in order to allow very low bit-rate transmission of the animation parameters. The proposed algorithm is based on a comprehensive analysis of the cross-correlation properties that characterize FAPs, which is here reported and discussed extensively. Based on this analysis, a subset of ten almost independent FAPs has been selected from the full set of 66 low-level FAPs to be transmitted and used at the decoder to interpolate the remaining ones. The performance achievable through the proposed algorithm have been evaluated objectively by means of conventional PSNR measures and compared to an alternative solution based on the increase of the quantization scale factor used for FAP encoding. The subjective evaluation and comparison of the results has also been made possible by uploading mpg movies on a freely accessible Web site. Experimental results demonstrate that the proposed FAP interpolation algorithm allows efficient parameter encoding at around 70 bits/frame or, in other words, at less than 2 kbits/s for smooth synthetic video at 25 frames/s.