Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies

Abstract

In this paper, we explore the possibility to gather perceptual impressions of prosodic prominence by exploiting the strong prosody-gesture link, i.e., by having listeners transform a perceptual impression into a motor movement, namely drumming, for two domains of prominence: word-level and syllable-level. A feasibility study reveals that such a procedure is indeed easily and speedily mastered by naïve listeners, but more difficult for word-level prominences. We furthermore examine whether “drummed” annotations are comparable to those gathered with more established annotation protocols based on cumulative naïve impressions and fine-grained expert ratings. These comparisons reveal high correspondences across all prominence annotation protocols, thus corroborating the general usefulness of the gestural approach. The analyses also reveal that all annotation protocols are strongly driven by structural linguistic considerations. We then use Random Forest Models to investigate the relative impact of signal and structural cues to prominence annotations. We find that expert ratings of prosodic prominence are guided comparatively more by structural concerns than those of naïve annotators, that word-level annotations are influenced more by structural linguistic cues than syllable-level ones, and that “drummed” annotations are driven least by structural cues. Lastly, we isolate two main listener strategies among our group of “drummers”, namely those integrating structural and signal cues to prominence, and those being guided predominantly by signal cues.