Factors affecting the integration of auditory and visual information in speech: The level effect

Abstract

The “McGurk effect” is a phenomenon in which an illusory syllable is perceived when discrepant auditory and visual information are combined. For example, an illusory /da/ is perceived when an auditory syllable /ba/ is paired with a video display of a talker saying /ga/. In a series of studies reported here, it was found that the level of the auditory signal substantially affects the illusion. Moreover, this effect of auditory level works in a direction that is counterintuitive. One might have postulated that as the level of the auditory signal is increased, making the auditory signal more pronounced, the illusion would decrease. These results indicate just the opposite. As the level of the auditory signal goes up, rather than it outweighing the visual signal and thereby decreasing the illusion, the effect is to increase the number of illusory responses. Paired were the same auditory syllable /ba/ at three levels, soft (45 dB SPL), moderate (58 dB SPL), and loud (66 dB SPL), with a video display of /ga/. Significant increases in illusory /da/ responses occurred as the auditory signal increased from soft to moderate and again from moderate to loud. The work thus shows that auditory level is an important determinant in the perception of the illusion. The paradox of the situation is that as the auditory signal increases in strength, the resulting percept becomes less veridical rather than more so. [Work supported by NIH.]