Mouth reversal extinguishes mismatch negativity induced by the McGurk illusion

Abstract

The sight of articulatory mouth movements (visual speech) influences auditory speech perception. This is demonstrated by the McGurk illusion in which incongruent visual speech alters the auditory phonetic percept. In behavioral studies, reversal of the vertical mouth direction has been reported to greatly reduce the McGurk illusion (Rosenblum et al., 2000). Such findings support the idea that audiovisual integration in speech to some extent relies on information regarding facial configuration. Here we ask whether this behavioral effect is reflected in a difference in neural activity in the auditory cortex. Mismatch Negativity (MMN) is a component in the auditory Event-Related Potential (ERP) that is elicited by a change in the auditory percept. It has been shown that the McGurk illusion can induce a MMN. We conducted an experiment in which the MMN could be elicited by the McGurk illusion induced by visual speech with either upright (unaltered) or vertically reversed mouth area. In a preliminary analysis, we found a Mismatch Negativity component induced by the McGurk illusion for 6 of 17 participants at electrode Cz when the mouth area was upright. In comparison, these participants produced no Mismatch Negativity when the mouth was reversed. These findings mirrored behavioral findings in the same subjects of a strong McGurk response for normal audiovisual speech, which was greatly reduced for stimuli with reversed mouth area.