Early stages of audiovisual speech processing—a magnetoencephalography study

Abstract

Speech is a multimodal process integrating auditory and visual information. It is still unsettled at which stage of processing visual and auditory data structures merge into a common percept. This study investigates the time domain of auditory M50 and M100 fields in response to audiovisual speech (whole-head MEG, 25 subjects). As in the original video recordings (/pa/ and /ta/), the visual articulatory cues preceded the acoustic signal (synthetic syllable) by ca. 150 ms. Visual nonspeech, acoustic nonspeech, visual-only, and acoustic-only stimuli served as control conditions. In accordance with previous studies, the M50 was consistently attenuated by visual motion cues, and a similar visual ‘‘negativation’’ effect could be observed even when no acoustic signal was played. As concerns the M100 field, visual nonspeech stimuli had an attenuating effect (only in the presence of an acoustic signal) whereas visual speech gave rise to hypoadditive M100 enhancement. Application of a six-dipole model showed a categorical-like visual /p/ versus /t/ effect. Since the source of this effect was located within insular cortex rather than the auditory system, we may assume that at the time of M100 the visual phonetic information has not yet been integrated into a common auditory/phonetic representation.