MPEG-4 Facial Animation Parameters Research Articles

3D facial model reconstruction, expressions synthesis and animation using single frontal face image

With better understanding of face anatomy and technical advances in computer graphics, 3D face synthesis has become one of the most active research fields for many human-machine applications, ranging from immersive telecommunication to the video games industry. In this paper we proposed a method that automatically extracts features like eyes, mouth, eyebrows and nose from the given frontal face image. Then a generic 3D face model is superimposed onto the face in accordance with the extracted facial features in order to fit the input face image by transforming the vertex topology of the generic face model. The 3D-specific face can finally be synthesized by texturing the individualized face model. Once the model is ready six basic facial expressions are generated with the help of MPEG-4 facial animation parameters. To generate transitions between these facial expressions we use 3D shape morphing between the corresponding face models and blend the corresponding textures. Novelty of our method is automatic generation of 3D model and synthesis face with different expressions from frontal neutral face image. Our method has the advantage that it is fully automatic, robust, fast and can generate various views of face by rotation of 3D model. It can be used in a variety of applications for which the accuracy of depth is not critical such as games, avatars, face recognition. We have tested and evaluated our system using standard database namely, BU-3DFE.

Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis [Social Sciences

The face conveys information about a person's age, sex, background, and identity; what they are feeling, thinking, or likely to do next. Facial expression regulates face-to-face interactions, indicates reciprocity and interpersonal attraction or repulsion, and enables intersubjectivity between members of different cultures. Facial expression indexes neurological and psychiatric functioning and reveals personality and socioemotional development. Not surprisingly, the face has been of keen interest to behavioral scientists. About 15 years ago, computer scientists became increasingly interested in the use of computer vision and graphics to automatically analyze and synthesize facial expression. This effort was made possible in part by the development in psychology of detailed coding systems for describing facial actions and their relation to basic emotions, that is, emotions that are interpreted similarly in diverse cultures. The most detailed of these systems, the Facial Action Coding System (FACS), informed the development of the MPEG-4 facial animation parameters for video transmission and enabled progress toward automated measurement and synthesis of facial actions for research in affective computing, social signal processing, and behavioral science. This article reports key advances in behavioral science that are becoming possible through these developments. Before beginning, automated facial image analysis and synthesis (AFAS) is briefly described.

Dynamic Facial Expression Analysis and Synthesis With MPEG-4 Facial Animation Parameters

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> This paper describes a probabilistic framework for faithful reproduction of dynamic facial expressions on a synthetic face model with MPEG-4 facial animation parameters (FAPs) while achieving very low bitrate in data transmission. The framework consists of a coupled Bayesian network (BN) to unify the facial expression analysis and synthesis into one coherent structure. At the analysis end, we cast the FAPs and facial action coding system (FACS) into a dynamic Bayesian network (DBN) to account for uncertainties in FAP extraction and to model the dynamic evolution of facial expressions. At the synthesizer, a static BN reconstructs the FAPs and their intensity. The two BNs are connected statically through a data stream link. Using the coupled BN to analyze and synthesize the dynamic facial expressions is the major novelty of this work. The novelty brings about several benefits. First, very low bitrate (9 bytes per frame) in data transmission can be achieved. Second, a facial expression is inferred through both spatial and temporal inference so that the perceptual quality of animation is less affected by the misdetected FAPs. Third, more realistic looking facial expressions can be reproduced by modelling the dynamics of human expressions. </para>

Towards modeling embodied conversational agent character profiles using appraisal theory predictions in expression synthesis

Appraisal theories in psychology study facial expressions in order to deduct information regarding the underlying emotion elicitation processes. Scherer's component process model provides predictions regarding particular face muscle deformations that are attributed as reactions to the cognitive appraisal stimuli in the study of emotion episodes. In the current work, MPEG-4 facial animation parameters are used in order to evaluate these theoretical predictions for intermediate and final expressions of a given emotion episode. We manipulate parameters such as intensity and temporal evolution of synthesized facial expressions. In emotion episodes originating from identical stimuli, by varying the cognitive appraisals of the stimuli and mapping them to different expression intensities and timings, various behavioral patterns can be generated and thus different agent character profiles can be defined. The results of the synthesis process are consequently applied to Embodied Conversational Agents (ECAs), aiming to render their interaction with humans, or other ECAs, more affective.

MPEG-4 facial expression synthesis

The current work will describe an approach to synthesize expressions, including intermediate ones, via the tools provided in the MPEG-4 standard based on real measurements and on universally accepted assumptions of their meaning, taking into account results of Whissel's study. Additionally, MPEG-4 facial animation parameters are used in order to evaluate theoretical predictions for intermediate expressions of a given emotion episode, based on Scherer's appraisal theory. MPEG-4 FAPs and action units are combined in modeling the effects of appraisal checks on facial expressions and temporal evolution issues of facial expressions are investigated. The results of the synthesizing process can then be applied to Embodied Conversational Agents (ECAs), rendering their interaction with humans, or other ECAs, more affective.

From Greta's mind to her face: modelling the dynamics of affective states in a conversational embodied agent

This paper describes the results of a research project aimed at implementing a ‘realistic’ 3D Embodied Agent that can be animated in real-time and is ‘believable and expressive’: that is, able to coherently communicate complex information through the combination and the tight synchronisation of verbal and nonverbal signals. We describe, in particular, how we ‘animate’ this Agent (that we called Greta) so as to enable her to manifest the affective states that are dynamically activated and de-activated in her mind during the dialog with the user. The system is made up of three tightly interrelated components: • A representation of the Agent Mind: this includes long and short-term affective components (personality and emotions) and simulates how emotions are triggered and decay over time according to the Agent's personality and to the context, and how several emotions may overlap. Dynamic belief networks with weighting of goals is the formalism we employ to this purpose. • A mark-up language to denote the communicative meanings that may be associated with dialog moves performed by the Agent. • A translation of the Agent's tagged move into a face expression, that combines appropriately the available channels (gaze direction, eyebrow shape, head direction and movement etc). The final output is a 3-D facial model that respects the MPEG-4 standard and uses MPEG-4 Facial Animation Parameters to produce facial expressions. Throughout the paper, we illustrate the results obtained, with an example of dialog in the domain of ‘Advice about eating disorders’. The paper concludes with an analysis of advantages of our cognitive model of emotion triggering and of the problems found in testing it. Although we did not yet complete a formal evaluation of our system, we briefly describe how we plan to assess the agent's believability in terms of consistency of its communicative behaviour.

A modular approach to facial feature segmentation on real sequences

In this paper a modular approach of gradual confidence for facial feature extraction over real video frames is presented. The problem is being dealt under general imaging conditions and soft presumptions. The proposed methodology copes with large variations in the appearance of diverse subjects, as well as of the same subject in various instances within real video sequences. Areas of the face that statistically seem to be outstanding form an initial set of regions that are likely to include information about the features of interest. Enhancement of these regions produces closed objects, which reveal—through the use of a fuzzy system—a dominant angle, i.e. the facial rotation angle . The object set is restricted using the dominant angle. An exhaustive search is performed among all candidate objects, matching a pattern that models the relative position of the eyes and the mouth. Labeling of the winner features can be used to evaluate the features extracted and provide feedback in an iterative framework. A subset of the MPEG-4 facial definition or facial animation parameter set can be obtained. This gradual feature revelation is performed under optimization for each step, producing a posteriori knowledge about the face and leading to a step-by-step visualization of the features in search.

Real-time MPEG-4 facial animation with 3D scalable meshes

In this paper the complete framework of a software package for real-time animation of 3D facial models according to the MPEG-4 specifications is described. The focus is mainly on the algorithmic solution that has been adopted for scaling the polygon mesh to a level of complexity suited to the specific hw/sw profile of the platform that hosts the decoder. The basic functionality of this system is its ability to deform virtual faces in response to MPEG-4 Facial Animation Parameters (FAP) according to “animation rules” whose computation is based on very simple “semantic” information previously associated with the model. The software architecture is modular and each module, implementing a specific functionality, can be easily plugged in or unplugged. In this paper, we begin with a general presentation of the system and then we present a brief description of its main modules: the face model, the mesh animation, the mesh calibration, the mesh simplification and the FAP decoding. While for the other modules a deeper analysis is left to referenced bibliography previously published by the authors, a comprehensive description is given here of the mesh simplification block together with experimental evidence of its efficiency. The algorithm we have adopted for mesh simplification is based on iterative edge contraction with the introduction of specific geometry constraints due to the particular nature of the 3D mesh representing an MPEG-4 animated face. In this paper it will be shown how rather complex 3D facial models, optionally textured, can be simplified in real-time in order to adapt the rendering performance to the hardware capabilities, whilst still maintaining compliance with the MPEG-4 specifications on facial animation.

An efficient use of MPEG-4 FAP interpolation for facial animation at 70 bits/frame

An efficient algorithm is proposed to exploit the facial animation parameter (FAP) interpolation modality specified by the MPEG-4 standard in order to allow very low bit-rate transmission of the animation parameters. The proposed algorithm is based on a comprehensive analysis of the cross-correlation properties that characterize FAPs, which is here reported and discussed extensively. Based on this analysis, a subset of ten almost independent FAPs has been selected from the full set of 66 low-level FAPs to be transmitted and used at the decoder to interpolate the remaining ones. The performance achievable through the proposed algorithm have been evaluated objectively by means of conventional PSNR measures and compared to an alternative solution based on the increase of the quantization scale factor used for FAP encoding. The subjective evaluation and comparison of the results has also been made possible by uploading mpg movies on a freely accessible Web site. Experimental results demonstrate that the proposed FAP interpolation algorithm allows efficient parameter encoding at around 70 bits/frame or, in other words, at less than 2 kbits/s for smooth synthetic video at 25 frames/s.

Digital watermarking of MPEG-4 facial animation parameters

The enforcement of intellectual property rights is difficult for digital data. One possibility of supporting enforcement is the embedding of digital watermarks containing information about copyright owner and/or receiver of the data. Watermarking methods have already been presented for audio, images, video, and polygonal 3D models. In this paper, we present a method for digital watermarking of MPEG-4 facial animation parameter data sets. The watermarks are additively embedded into the parameter values and can be retrieved either from the watermarked parameters or from video sequences rendered using the watermarked animation parameters for head animation. For retrieval from rendered sequences, the facial animation parameters have to be estimated first. We use a model-based approach for the estimation of the facial parameters that combines a motion model of an explicit 3D textured wireframe with the optical flow constraint from the video data. This leads to a linear algorithm that is robustly solved in a hierarchical framework with low computational complexity. Experimental results confirm the applicability of the presented watermarking technique.