Chapter 9. Multimodal story-retelling

Abstract

Cognitive load has been shown to influence speakers’ gesture production, but how it effects the duration, form, and functions of gestures is not yet well understood. This study examines how increased cognitive load influenced the production of co‑speech and co‑thought gestures by second language (L2) learners, and further looks at how different types of gestures correspond to specific functions. Twenty-four English as an L2 participants were recruited and required to watch a pre-determined cartoon and then retell the story in English. Participants were divided into two cognitive load conditions for the story retelling: one group was given a task structure for story retelling, while the other was not. The results of this study show that enhancing cognitive load significantly increased the frequency of both co‑speech and co‑thought gestures and the duration of co‑speech gestures, but not co‑thought gestures. With regard to the correspondence of gesture types and their functions for conceptualization (i.e., activation, manipulation, packaging, exploration), this study found co‑speech and co‑thought gestures serve different functions in L2 narrative tasks. Activation is dominant in every type of co‑speech gesture, especially in iconic and metaphoric gestures. Deictic gestures mostly function for packaging and exploration and co‑thought gestures for exploration and manipulation. This study provides necessary clarification on the relationship between cognitive load and gestures, including novel interactions of gesture duration and functions of gestures for conceptualization.