Creating 3D speech-driven talking heads: a probabilistic network approach

Abstract

We present a probabilistic approach to decide whether or not extracted facial features are appropriate for creating 3D face models. Automatically extracted 2D facial features from a video sequence are fed into the proposed probabilistic framework before a corresponding 3D face model is built to avoid generating unnatural or non-realistic 3D faces. In addition, a new algorithm for audio-to-visual conversion based on constrained optimization is presented to generate visual parameters for driving the mouth movement of the 3D face models from speech. Lagrangian optimization is applied to transform a constrained problem into an unconstrained problem. Experimental results are provided to show the effectiveness and validity of the proposed algorithms for various video sequences and speech.