Abstract
The McGurk effect, an incongruent pairing of visual /ga/–acoustic /ba/, creates a fusion illusion /da/ and is the cornerstone of research in audiovisual speech perception. Combination illusions occur given reversal of the input modalities—auditory /ga/-visual /ba/, and percept /bga/. A robust literature shows that fusion illusions in an oddball paradigm evoke a mismatch negativity (MMN) in the auditory cortex, in absence of changes to acoustic stimuli. We compared fusion and combination illusions in a passive oddball paradigm to further examine the influence of visual and auditory aspects of incongruent speech stimuli on the audiovisual MMN. Participants viewed videos under two audiovisual illusion conditions: fusion with visual aspect of the stimulus changing, and combination with auditory aspect of the stimulus changing, as well as two unimodal auditory- and visual-only conditions. Fusion and combination deviants exerted similar influence in generating congruency predictions with significant differences between standards and deviants in the N100 time window. Presence of the MMN in early and late time windows differentiated fusion from combination deviants. When the visual signal changes, a new percept is created, but when the visual is held constant and the auditory changes, the response is suppressed, evoking a later MMN. In alignment with models of predictive processing in audiovisual speech perception, we interpreted our results to indicate that visual information can both predict and suppress auditory speech perception.
Highlights
The cornerstone of research into the audiovisual nature of speech perception is the McGurk effect, an illusion in which visual articulatory information alters speech perception [1]
There were no significant differences in the number of usable responses to standards across conditions (F (3, 72) = 0.10, p = 0.960, η2 = 0.004), with means and standard deviations as follows: auditory steady–visual change (AsVc) (M = 201, SD = 59.1), visual change steady–auditory (VsAc) (M = 202, SD = 51.8), auditory only (M = 200, SD = 72.7), and visual only (M = 192, SD = 65.2)
We propose that in the VsAc condition, visual predictions constrain and suppress the earlier, automatic response that was seen in the AsVc and auditory-only conditions, and a process of comparison as reflected by the mismatch negativity (MMN) in the later time window
Summary
The cornerstone of research into the audiovisual nature of speech perception is the McGurk effect, an illusion in which visual articulatory information alters speech perception [1]. /ba/, deriving the fused percept alveolar /da/. One interpretation of the effect of visual articulatory information on speech perception is that as the visual signal naturally precedes the auditory signal, it provides predictive information that constrains the possible identities of the upcoming auditory speech sound [2,3,4,5]. When the McGurk illusion occurs, fusing aspects of conflicting modalities, the derived percept is one that is categorically different from the given input [6]. Fusion responses are dependent upon the ambiguity in the signal, such that the visual aspect of the stimulus overrides the auditory percept
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
