Kinematic properties of concurrently recorded speech and body gestures and their relationship to prosodic structure

Abstract

Previous research has shown that body gesturing, such as hand and torso movements, is related to prosodic structure (e.g., Esteve-Gibert & Prieto 2013). However, the details of the relationship are not well understood and specifically the effects of prosodic structure on gesturing duration have received little attention. We present an experiment investigating the effects of stress and prosodic boundaries on the temporal properties of body gesturing and gestures of the vocal tract. Two speakers read 6 repetitions of 12 sentences examining the effect of boundary (no boundary, ip boundary, IP boundary), stress (no stress, stress) and their interaction, phrase finally and phrase initially (144 sentences total). They were observed through concurrent recordings of speech audio, vocal tract gestures using electromagnetic articulometry, and body gesturing using Vicon motion capture. We test the hypothesis that manual movements lengthen under prominence and at boundaries and that the lengthening at boundaries is cumulative, parallel to the behavior of gestures of the vocal tract. Analyzing speech prosody and body gesturing in this manner will allow us to investigate whether prosodic control extends to body gesturing and to identify coordinative patterns that are relevant to fields such as language pedagogy and speech pathology. [Work supported by NIH.]