Expressive Modulation of Neutral Visual Speech

Abstract

Expressive speech animation, or expressive visual speech, is the process by which some representation of facial features (graphical, statistical, or otherwise) is smoothly changed over time to articulate verbal speech, convey emotion through facial expression, or express other characteristics such as visual prosody. Used extensively in the production of movies and computer games, current techniques for visual speech animation are slow, laborious, and expensive. Therefore, efforts to automate the complex processes involved are of great interest to these multibillion-dollar industries. In this article, the authors describe a method for transforming speech animation between different emotional expressions. When the same sentence is spoken in two different expressive styles, a large proportion of the two sentences is the same. Intuitively, then, after factoring out timing differences, the residual is the expression. Based on earlier work showing two general categories of facial movements in expressive speech--high-frequency speech components (the content) and low-frequency expressive components (the style)--the authors use independent component analysis (ICA) to factorize these movements and show how the energy for different speaking styles is distributed in this space. They transform speaking style by projecting novel animation curves into the low-dimensional ICA space, redistributing the energy in the independent components, and reconstructing the animation by inverting the projection. As they describe, a single ICA model can be used for factoring multiple expressive styles, and their method works on a variety of data types. Evaluations show that viewers can identify the expressive style generated and have difficulty distinguishing transformed animation from ground truth. Finally, they show how their technique can be used to represent complex blends of expression.