Synchrony Perception and Inverse Effectiveness: Are they Complementary or Contrasting in Audiovisual Speech Integration?

Abstract

Inverse effectiveness, one of the three principles of multisensory integration, was formulated in the context of single neuron recordings and, according to it multisensory stimuli are more likely, robustly or strongly integrated when the most effective unisensory response is relatively weak (Meredith & Stein, 1983). This principle has widely been applied in order to interpret behavioral data (Holmes, 2007). Another principle of multisensory integration is synchrony perception (Spence & Squire, 2003). That is, our ability to integrate the fragmented neural time of multiple sensory inputs in an illusory synchronous percept. This percept is defined by a temporal window of integration, where the perceptual system accepts asynchronies in multisensory events while maintaining the subjective impression of synchrony (Vatakis & Spence, 2010). We aim to clarify the role of the two principles – do they complement or contrast each other in multisensory integration? In order to address this question, two experiments will be conducted. In Experiment 1, we will assess the influence of audiovisual ambiguity as noise on speech perception by presenting brief syllables with various levels of auditory, visual, and audiovisual ambiguity. This will allow us to define the level of multisensory gain as a function of ambiguity. In Experiment 2, we use the stimuli exhibiting the highest and lowest level of multisensory gain obtained in Exp. 1 and present them at various levels of stimulus onset asynchronies in a simultaneity judgment task. If the two principles of multisensory integration complement each other, then we expect that smaller temporal windows will be observed in the case of greater multisensory gain: The increased gain due to stronger multisensory integration modulates the temporal processing of audiovisual speech stimuli, since the unisensory counterparts are strongly bound. If they contrast each other then we expect that for the high-gain stimuli the temporal window will be larger as compared to the low-gain stimuli, given that high ambiguity in both sensory inputs (thus, smaller predictive power for the visual and auditory inputs; Halle, 2002; van Wassenhove et al., 2005; Vatakis et al., 2012) will lead to larger processing times.