Mechanisms of associative word learning: Benefits from the visual modality and synchrony of labeled objects

Abstract

Despite humans' ability to communicate about concepts relating to different senses, word learning research tends to largely focus on labeling visual objects. Although sensory modality is known to influence memory and learning, its specific role for word learning remains largely unclear. We investigated associative word learning in adults, that is the association of an object with its label, by means of event-related brain potentials (ERPs). We evaluated how learning is affected by object modality (auditory vs visual) and temporal synchrony of object-label presentations (sequential vs simultaneous). Across 4 experiments, adults were, in training phases, presented either visual objects (real-world images) or auditory objects (environmental sounds) in temporal synchrony with or followed by novel pseudowords (2 × 2 design). Objects and pseudowords were paired either in a consistent or an inconsistent manner. In subsequent testing phases, the consistent pairs were presented in matching or violated pairings. Here, behavioral and ERP responses should reveal whether consistent object–pseudoword pairs had been successfully associated with one another during training. The visual-object experiments yielded behavioral learning effects and an increased N400 amplitude for violated versus matched pairings indicating short-term retention of object–word associations, in both the simultaneous and sequential presentation conditions. For the auditory-object experiments, only the simultaneous, but not the sequential presentation, revealed similar results. Across all experiments, we found behavioral and ERP correlates of associative word learning to be affected by both sensory modality and partly, by temporal synchrony of object-label combinations. Based on our findings, we argue for independent advantages of temporal synchrony and visual modality in associative word learning.