ExpressGesture: Expressive gesture generation from speech through database matching

Abstract

AbstractCo‐speech gestures are a vital ingredient in making virtual agents more human‐like and engaging. Automatically generated gestures based on speech‐input often lack realistic and defined gesture form. We present a database‐driven approach guaranteeing defined gesture form. We built a large corpus of over 23,000 motion‐captured co‐speech gestures and select individual gestures based on expressive gesture characteristics that can be estimated from speech audio. The expressive parameters are gesture velocity and acceleration, gesture size, arm swivel, and finger extension. Individual, parameter‐matched gestures are then combined into animated sequences. We evaluate our gesture generation system in two perceptual studies. The first study compares our method to the ground truth gestures as well as mismatched gestures. The second study compares our method to five current generative machine learning models. Our method outperformed mismatched gesture selection in the first study and showed competitive performance in the second.