The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect.

Jeroen J Stekelenburg,Mirjam Keetels

doi:10.1007/s00221-015-4363-0

Abstract

The Colavita effect refers to the phenomenon that when confronted with an audiovisual stimulus, observers report more often to have perceived the visual than the auditory component. The Colavita effect depends on low-level stimulus factors such as spatial and temporal proximity between the unimodal signals. Here, we examined whether the Colavita effect is modulated by synesthetic congruency between visual size and auditory pitch. If the Colavita effect depends on synesthetic congruency, we expect a larger Colavita effect for synesthetically congruent size/pitch (large visual stimulus/low-pitched tone; small visual stimulus/high-pitched tone) than synesthetically incongruent (large visual stimulus/high-pitched tone; small visual stimulus/low-pitched tone) combinations. Participants had to identify stimulus type (visual, auditory or audiovisual). The study replicated the Colavita effect because participants reported more often the visual than auditory component of the audiovisual stimuli. Synesthetic congruency had, however, no effect on the magnitude of the Colavita effect. EEG recordings to congruent and incongruent audiovisual pairings showed a late frontal congruency effect at 400–550 ms and an occipitoparietal effect at 690–800 ms with neural sources in the anterior cingulate and premotor cortex for the 400- to 550-ms window and premotor cortex, inferior parietal lobule and the posterior middle temporal gyrus for the 690- to 800-ms window. The electrophysiological data show that synesthetic congruency was probably detected in a processing stage subsequent to the Colavita effect. We conclude that—in a modality detection task—the Colavita effect can be modulated by low-level structural factors but not by higher-order associations between auditory and visual inputs.

Highlights

It is well established that for many multisensory events one sensory modality dominates the other
When participants made an erroneous response in the audiovisual trials, they reported more frequently to have perceived a visual stimulus than an auditory stimulus
It was expected that synesthetic congruency would modulate the magnitude of the Colavita effect because synesthetic congruency has been shown to affect other manifestations of audiovisual integration (Parise and Spence 2008, 2009; Bien et al 2012), while at the same time the Colavita effect is sensitive to factors that are critical for audiovisual integration (Koppen and Spence 2007a, c)

Summary

Introduction

It is well established that for many multisensory events one sensory modality dominates the other. Several studies show that synesthetic congruency modulates multisensory integration In these studies, synesthetic congruency between visual size and auditory pitch affected the spatial ventriloquist effect (Parise and Spence 2009; Bien et al 2012), and audiovisual temporal order judgment (TOJ) (Parise and Spence 2009). With increasing strength of the unity assumption, there is a higher chance that the visual stimulus adequately describes the AV stimulus, thereby eclipsing the auditory stimulus and making the auditory percept redundant (Koppen and Spence 2007a) It is, not self-evident that there will be an effect of synesthetic congruency because the size of the Colavita effect is modulated by structural factors that are critical in multisensory integration, effects of semantic congruency on the Colavita effect are less consistent (Koppen et al 2008; Stubblefield et al.2013). By contrasting event-related potentials (ERPs) of the synesthetic congruent and incongruent audiovisual stimuli, we examined at what stage of perception synesthetic congruency is processed (cf. Bien et al 2012)

Participants

Results

Discussion