Localizing, describing, interpreting: effects of different audio text structures on attributing meaning to digital pictures

Abstract

Based on previous research on multimedia learning and text comprehension, an eye-tracking study was conducted to examine the influence of audio text coherence on visual attention and memory in a multimedia learning situation with a focus on picture comprehension. Audio text coherence was manipulated by the type of LDI structure, that is, whether localization, description, and interpretation followed in immediate succession for each pictorial detail or whether localizations and description of details were separated from their interpretation. Results show that with a LDI integrated structure compared to a LDI separated structure the referred-to picture elements were fixated longer during interpretation parts, and linkages between descriptions and interpretations were better recalled and recognized. The effects on recall and recognition of linkages were fully mediated by fixation times. This pattern of results can be explained by an interplay between audio text coherence and dual coding processes. It points out the importance of local coherence and the provision of localization information in audio explanations as well as visual attention to allow for dual coding processes that can be used to better attribute meaning to picture details. Practical implications for the design of educational videos, audio texts on websites, and audio guides are discussed.