Three‐dimensional (3‐D) facial model‐based description and synthesis of facial expressions

Abstract

AbstractThe automatic synthesis of the facial expression by computer is one of the most basic techniques to be applied in various fields. This paper describes a method to synthesize various facial expressions by modification of the 3‐D model of the face. The 3‐D facial model is obtained by adjusting the prepared 3‐D facial shape model to the frontal view of the object neutral face image, and by projecting the gray‐level informations.This paper proposes first a hierarchical model of facial description units for the hierarchical description of the facial expressions. The action unit (AU) of the FACS (facial action coding system), which is known as a systematic method of describing the facial expression, is considered as a layer in the proposed hierarchical model. AU of FACS is realized on a computer, and the automatic synthesis of the expression is executed.This procedure is based on the muscular structure of the face and allows cancellation of individual differences. At present, 34 kinds of AUs are implemented. The graylevel information for wrinkles and teeth, which are important in the synthesis of expressions, is not contained in the 3‐D facial model obtained from the neutral expression. Thus, such information is extracted from other face images, and registered as auxiliary information of the 3‐D facial model. This leads to synthesis of natural expressions. Finally, numerous examples for the facial expression synthesis are presented to show the usefulness of the method.