Abstract
Visual narratives communicate event sequences by using different code systems such as pictures and texts. Thus, comprehenders must integrate information from different codalities. This study addressed such cross-codal integration processes by investigating how the codality of bridging-event information (i.e., pictures, text) affects the understanding of visual narrative events. In Experiment 1, bridging-event information was either present (as picture or text) or absent (i.e., not shown). The viewing times for the subsequent picture depicting the end state of the action were comparable within the absent and the text conditions. Further, the viewing times for the end-state picture were significantly longer in the text condition as compared to the pictorial condition. In Experiment 2, we tested whether replacing bridging-event information with a blank panel increases viewing times in a way similar to the text condition. Bridging event information was either present (as picture) or absent (not shown vs. blank panel). The results replicated Experiment 1. Additionally, the viewing times for the end-state pictures were longest in the blank condition. In Experiment 3, we investigated the costs related to integrating information from different codalities by directly comparing the text and picture conditions with the blank condition. The results showed that the distortion caused by the blank panel is larger than the distortion caused by cross-codal integration processes. Summarizing, we conclude that cross-codal information processing during narrative comprehension is possible but associated with additional mental effort. We discuss the results with regard to theories of narrative understanding.
Highlights
A central characteristic of visual narratives is that information is presented in different codalities
We did not observe a difference between the experimental conditions for the end state + 1 pictures, indicating that cross-codal integration does not extend beyond the end-state pictures
We propose that the present situation model is preserved and that the increased viewing times in the text condition reflect cross-codal integration costs resulting from recoding-interference
Summary
A central characteristic of visual narratives is that information is presented in different codalities (e.g., as pictures or a text; for an example see Fig. 1). We use the term codality, following the distinction between the modality and codality of information (Clark & Salomon, 1986). While the modality refers to the sensory channel through which the information to be conveyed is perceptually processed (e.g., visual or auditory), the codality refers to the specific information coding (e.g., verbal, pictorial, symbolic). An “apple” can be represented in a photograph (visual modality, pictorial code), be written (visual modality, verbal code), or be represented by an icon (visual modality, symbolic code). A picture story with isolated text elements is uni-modal (visual) but multi-codal
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
