Audiovisual Integration of Speech in a Bistable Illusion

Abstract

Visible speech enhances the intelligibility of auditory speech when listening conditions are poor [1], and can modify the perception of otherwise perfectly audible utterances [2]. This audiovisual perception is our most natural form of communication and one of our most common multisensory phenomena. However, where and in what form the visual and auditory representations interact is still not completely understood. Although there are longstanding proposals that multisensory integration occurs relatively late in the speech-processing sequence [3], considerable neurophysiological evidence suggests that audiovisual interactions can occur in the brain stem and primary sensory cortices [4, 5]. A difficulty testing such hypotheses is that when the degree of integration is manipulated experimentally, the visual and/or auditory stimulus conditions are drastically modified [6, 7]; thus, the perceptual processing within a modality and the corresponding processing loads are affected [8]. Here, we used a bistable speech stimulus to examine the conditions under which there is a visual influence on auditory perception in speech. The results indicate that visual influences on auditory speech processing, at least for the McGurk illusion, necessitate the conscious perception of the visual speech gestures, thus supporting the hypothesis that multisensory speech integration is not completed in early processing stages.