Sequences in harmony: Cognitive interactions between musical and visual narrative structure

Abstract

From film and television to graphic storytelling, tonal music can accompany visual narratives in a variety of contexts. The apprehension of both musical and narrative sequences involves temporal categories in ordered patterning, which raises an interesting question: Do musical progressions and visual narratives rely on shared sequence processing mechanisms? If this is the case, then cues from music and sequential static images, when presented simultaneously, should interact during audiovisual online processing. We tested this question by measuring reaction times to target picture panels appearing in visual narrative (comic strip) sequences, which were presented panel by panel and synchronized with musical chord progressions. Image sequences were either coherent narratives or incoherent (random) panels, and they were aligned with musical accompaniment consisting of coherent tonal chord progressions or non-tonal (unrelated) chords. Reaction times were faster for target images in coherent sequences than incoherent sequences, and even faster for coherent images with tonal accompaniment than non-tonal chords. This indicated an interaction between sequential structures across domains. We take these results as evidence for a shared, domain-general sequence processing mechanism operating across music and visual narrative.