Exploring the basis of the “McGurk effect”: Can perceivers combine information from a female face and a male voice?

Abstract

In the “McGurk” effect, observers typically report the illusory syllable /da/ when they hear the auditory syllable /ba/ presented in synchrony with a video display of a talker saying /ga/. In such experiments, there is usually congruence between the two modalities in that the same talker produces both the auditory and the visual signals. In the experiments reported here, the effect of reducing the congruence between the two modalities on the magnitude of the McGurk effect was examined. This was accomplished by dubbing a male talker's voice onto a video tape containing a female talker's face, and a female talker's voice onto a video tape containing a male talker's voice. These “cross-dubbed” video tapes were compared to normal video tapes in which the male talker's voice was dubbed onto a male talker's face, and the female talker's voice was dubbed onto a female talker's face. The results show that even though there was clear incompatibility in the talker characteristics between the auditory and visual signals for the cross-dubbed stimuli, there was little difference in the magnitude of the effect compared to the normal stimuli. These results indicate that the mechanism for integrating speech information from the two modalities is not sensitive to certain incompatibilities, even when they are perceptually apparent. [Work supported by NIH.]