Temporal characteristization of auditory-visual coupling in speech

Abstract

This work examines the coupling between the acoustic and visual components of speech as it evolves through time. Previous work has shown a consistent correspondence between face motion and spectral acoustics, and between fundamental frequency (F0) and rigid body motion of the head [Yehia et al. (2002), JPHON, 30, 555–568]. Although these correspondences have been estimated both for sentences and for running speech, the analyses have not taken into account the temporal structure of speech. As a result, the role of temporal organization in multi-modal speech cannot be estimated. The current study is a first effort to correct this deficit. We have developed an algorithm, based on recurrent correlation, that computes the correlation between measurement domains (e.g., head motion and F0) as a time-varying function. Using this method regions of high or low correlation, or of rapid transition (e.g., from high to low) can be associated with visual and auditory events. The analysis of the time-varying coupling of multi-modal events has implications for speech planning and synchronization between speaker and listener.