Cognitive and semiotic determinants of sign order in gestural and pictorial event representations

Abstract

The way people order signs in non-verbal event representations of events has been a topic of extensive research in recent decades, with conflicting findings. Based on a literature review, we distinguish the following factors that have been argued to influence sign order: (a) diagrammatic iconicity, (b) manipulation vs. construction events, (c) reversibility, (d) most common word order of L1, (e) the semiotic system (gesture, depiction) used, and (f) a putative “natural” Agent–Patient–Act order. To test the role of these factors, we conducted a study where Swedish participants observed events that varied with respect to reversibility and construction/manipulation status, and then had to communicate them to an addressee using gestures for half the stimuli. For the other half, they used sequences of simple drawings of the event participants and the action. The results showed the huge influence of the semiotic system (e) on sign order. There was a role of reversibility (c) only when gestures were used, while L1 word order (d) only had an effect when sequences of pictures were used. The “ontological status” of the Patient (b) was shown to affect the representations in both semiotic systems but was much stronger for gestures. Even the two most general factors (a) and (f) where shown to be sensitive to the type of semiotic system. Future studies of how such cognitive and semiotic determinants interact are needed to fully understand the phenomenon of sign order preference.