Bridging the gap between speech segmentation and word-to-world mappings: Evidence from an audiovisual statistical learning task

Abstract

How are adult second language learners able to segment words and map them to referents in the new language? The present study explores this unresolved issue by using a new multimodal learning paradigm that tracks the first steps in learning new words and their mappings to visual referents. It encompasses a continuous audiovisual stream in which transitional probability of syllables is the only acoustic cue available to segment the stream into words, and a visual stream of object images that accompanies the novel words. The objects are systematically varied in terms of constancy of word-picture association and meaningfulness. The results indicated good word-referent mapping and word segmentation after short exposure to the audiovisual stream. Mapping words with pictures was more effective when the visual referents were meaningful objects. In word segmentation, the consistency of the word-picture association affected segmentation performance. The effect of associative strength on segmentation performance was most prominent with meaningful objects, albeit associative strength did not interact significantly with meaningfulness. The present results suggest that word segmentation and word-referent mapping are closely related processes: word segmentation is affected by the consistency of the mapping relationship and both segmentation and mapping can be accomplished under the same short exposure.