A protocol for comparing gesture and prosodic boundaries in multimodal corpora

Abstract

This paper aims to summarize the key points and results of an analysis of the alignment of gesture and prosodic boundaries collected through manual annotation and automated treatment. For this purpose, we used a multimodal corpus of spontaneous speech, the BGEST corpus (Barros, 2021), part of the C-ORAL-BRASIL language resources (Moneglia and Cresti, 2005; Raso and Mello, 2012). Gestures and prosodic boundaries were manually annotated with ELAN. The gestural an-notation followed partly the Linguistic Annotation System for Gestures guidelines (Bressem et al., 2013), whereas the prosodic boundaries followed the perceptual criterion adopted in other C-ORAL resources. The tabular data outputted by ELAN was then analyzed by a script that compares the alignment of gesture phrases and intonation units. The results point to a larger gestural phrase that encompasses the intonation unit, while the stroke is almost always within the intonation unit. Around 85 percent of all information units align with a gesture. The analysis also points that the time values of alignment are shorter than analyzed by Loehr (2004) both for initial as well for final boundaries and more complex than proposed by Cantalini (2018). This indicates that more research is needed before we can set an approximate value for the overlap of larger units.