Speech timing prediction in multimodal human-computer interaction

Abstract

This paper proposes a quantitative model of natural modality integration for speech and pointing gestures. An experiment is described that study temporal synchronization between speech and pointing gestures during multimodal interaction. The end of a pointing gesture (MT) is shown to be synchronized with either the key word of an expression or the deictic marker of a deictic expression. It is also shown that MT tends to be slightly behind the beginning of its associated word; this tendency becoming more marked for lengthy expressions (more than 6 syllables). All correlation scores were found to be above 0.9.