Language Experience Changes Audiovisual Perception.

Viorica Marian,Tuan Lam,Sayuri Hayakawa,Scott Schroeder

doi:10.3390/brainsci8050085

Viorica Marian, Tuan Lam + Show 2 more

Open Access

https://doi.org/10.3390/brainsci8050085

Copy DOI

Abstract

Can experience change perception? Here, we examine whether language experience shapes the way individuals process auditory and visual information. We used the McGurk effect—the discovery that when people hear a speech sound (e.g., “ba”) and see a conflicting lip movement (e.g., “ga”), they recognize it as a completely new sound (e.g., “da”). This finding that the brain fuses input across auditory and visual modalities demonstrates that what we hear is profoundly influenced by what we see. We find that cross-modal integration is affected by language background, with bilinguals experiencing the McGurk effect more than monolinguals. This increased reliance on the visual channel is not due to decreased language proficiency, as the effect was observed even among highly proficient bilinguals. Instead, we propose that the challenges of learning and monitoring multiple languages have lasting consequences for how individuals process auditory and visual information.

Highlights

As we go about our day, our minds are constantly integrating incoming sensory inputs
There were significant main effects of AV status, but no effects of self-reported proficiency or interactions. These results provide a measure of proficiency and demonstrate that while late bilinguals experienced some deficit in comprehending speech under noisy conditions, early bilinguals and monolinguals performed well
We propose that beyond compensating for on-line difficulties in comprehending speech, early bilingual experience may fundamentally change how people attend to and subsequently perceive audiovisual information

Summary

Introduction

As we go about our day, our minds are constantly integrating incoming sensory inputs. In its extreme form, sensory integration can result in rare but fascinating synesthetic experiences such as seeing yellow when hearing “o” or tasting the month of June [1,2]. Non-human primates [4], speaking to how naturally and seemingly effortlessly we bind different sensory inputs. It might be counterintuitive to think that such a fundamental perceptual process could be influenced by our experiences. To explore this idea, we turn to language as a source of experience, and ask whether the use of two languages (i.e., bilingualism) modulates audio-visual integration. Can being bilingual change how and what we hear?

Objectives

Methods

Results

Discussion

Conclusion