Abstract
Everyday communication is enriched by the visual environment that listeners concomitantly link to the linguistic input. If and when visual cues are integrated into the mental meaning representation of the communicative setting, is still unclear. In our earlier findings, the integration of linguistic cues (i.e., topic-hood of a discourse referent) reduced discourse updating costs of the mental representation as indicated by reduced sentence-initial processing costs of the non-canonical word order in German. In the present study we tried to replicate our earlier findings by replacing the linguistic cue by a visual attention-capture cue presented below the threshold of perception in order to direct participant’s attention to a depicted referent. While this type of cue has previously been shown to modulate word order preferences in sentence production, we found no effects on sentence comprehension. We discuss possible theory-based reasons for the null effect of the implicit visual cue as well as methodological caveats and issues that should be considered in future research on multimodal meaning integration.
Highlights
Everyday communication is multimodal, comprising linguistic as well as extra-linguistic information
We raise the underlying question if the accessibility degree of mentally represented discourse referents is affected by this type of implicit visual cue or if this is limited to linguistic cues
3.1.1 event-related potentials (ERPs) results of sentence processing following the implicit visual cue For sentence-initial processing, statistical analyses in the time windows of 100 - 300 ms and 300 - 500 ms time-locked to the onset of DP1 neither revealed any statistically significant main effects of CUE (TOPIC vs. NEUTRAL) or WORD ORDER (SO vs. OS), nor significant interactions of CUE, WORD ORDER and/or ROI [p > .1]
Summary
Everyday communication is multimodal, comprising linguistic as well as extra-linguistic (e.g., visual) information. We differentiate between linguistic salience which is verbally induced by, for instance, subject-hood or topic-hood of a referent vs visual salience which is induced by, for instance, exogenous visual cues to a depicted referent. Explicit visual cues are presented above the threshold of perception (i.e., consciously) (for an overview about neuronal modulations by stimulus-driven (i.e., sensory cue-based) visual attention mechanisms, see Corbetta & Shulman, 2002). We aim to test if visual salience induced by an implicit visual attention-capture cue to a depicted referent impacts online sentence-initial processing in a similar way as it has previously been shown for linguistic salience (Burmester, Spalek, & Wartenburger, 2014). We raise the underlying question if the accessibility degree of mentally represented discourse referents is affected by this type of implicit visual cue or if this is limited to linguistic cues
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
