Animated Heads: From 3D Motion Fields to Action Descriptions

Abstract

We demonstrate a method to compute three-dimensional (3D) motion fields on a face. Twelve synchronized and calibrated cameras are positioned around a talking person, and observe its head in motion. We represent the head as a deformable mesh, which is fitted in a global optimization step to silhouette-contour and multi-camera stereo data derived from all images. The non-rigid displacement of the mesh from frame to frame, the 3D motion field, is determined from the spatiotemporal derivatives in all the images. We integrate these cues over time, thus producing an animated representation of the talking head. Our ability to estimate 3D motion fields points to a new framework for the study of action. The 3D motion fields can serve as an intermediate representation, which can be analyzed using geometrical and statistical tools for the purpose of extracting representations of generic human actions.KeywordsFacial ExpressionRigid MotionMotion FieldAction DescriptionVisual HullThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.