Temporal Ventriloquism Research Articles

Synesthetic congruency modulates the temporal ventriloquism effect

People sometimes find it easier to judge the temporal order in which two visual stimuli have been presented if one tone is presented before the first visual stimulus and a second tone is presented after the second visual stimulus. This enhancement of people's visual temporal sensitivity has been attributed to the temporal ventriloquism of the visual stimuli toward the temporally proximate sounds, resulting in an expansion of the perceived interval between the two visual events. In the present study, we demonstrate that the synesthetic congruency between the auditory and visual stimuli (in particular, between the relative pitch of the sounds and the relative size of the visual stimuli) can modulate the magnitude of this multisensory integration effect: The auditory capture of vision is larger for pairs of auditory and visual stimuli that are synesthetically congruent than for pairs of stimuli that are synesthetically incongruent, as reflected by participants' increased sensitivity in discriminating the temporal order of the visual stimuli. These results provide the first evidence that multisensory temporal integration can be affected by the synesthetic congruency between the auditory and visual stimuli that happen to be presented.

Tactile-visual temporal ventriloquism: No effect of spatial disparity

Participants made visual temporal order judgments (TOJs) about which of two lights appeared first while task-irrelevant vibrotactile stimuli delivered to the index finger were presented before the first and after the second light. Temporally misaligned tactile stimuli captured the onsets of the lights, thereby improving sensitivity on the visual TOJ task, indicative of tactile-visual (TV) temporal ventriloquism (Experiment 1). The size of this effect was comparable to auditory-visual (AV) temporal ventriloquism (Experiment 2). Spatial discordance between the TV stimuli, as in the AV case, did not harm the effect (Experiments 3 and 4). TV stimuli thus behaved like AV stimuli, demonstrating that spatial co-occurrence is not a necessary constraint for intersensory pairing to occur.

The Effect of Brief Auditory Stimuli on Visual Apparent Motion

When two discrete stimuli are presented in rapid succession, observers typically report a movement of the lead stimulus toward the lag stimulus. The object of this study was to investigate crossmodal effects of irrelevant sounds on this illusion of visual apparent motion. Observers were presented with two visual stimuli that were temporally separated by interstimulus onset intervals from 0 to 350 ms. After each trial, observers classified their impression of the stimuli using a categorisation system. The presentation of short sounds intervening between the visual stimuli facilitated the impression of apparent motion relative to baseline (visual stimuli without sounds), whereas sounds presented before the first and after the second visual stimulus as well as simultaneously presented sounds reduced the motion impression. The results demonstrate an effect of the temporal structure of irrelevant sounds on visual apparent motion that is discussed in light of a related multisensory phenomenon, 'temporal ventriloquism', on the assumption that sounds can attract lights in the temporal dimension.

Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism

The authors examined how principles of auditory grouping relate to intersensory pairing. Two sounds that normally enhance sensitivity on a visual temporal order judgement task (i.e. temporal ventriloquism) were embedded in a sequence of flanker sounds which either had the same or different frequency (Exp. 1), rhythm (Exp. 2), or location (Exp. 3). In all experiments, we found that temporal ventriloquism only occurred when the two capture sounds differed from the flankers, demonstrating that grouping of the sounds in the auditory stream took priority over intersensory pairing. By combining principles of auditory grouping with intersensory pairing, we demonstrate that capture sounds were, counter-intuitively, more effective when their locations differed from that of the lights rather than when they came from the same position as the lights.

Evaluating the influence of the ‘unity assumption’ on the temporal perception of realistic audiovisual stimuli

Vatakis, A. and Spence, C. (in press) [Crossmodal binding: Evaluating the ‘unity assumption’ using audiovisual speech stimuli. Perception &Psychophysics] recently demonstrated that when two briefly presented speech signals (one auditory and the other visual) refer to the same audiovisual speech event, people find it harder to judge their temporal order than when they refer to different speech events. Vatakis and Spence argued that the ‘unity assumption’ facilitated crossmodal binding on the former (matching) trials by means of a process of temporal ventriloquism. In the present study, we investigated whether the ‘unity assumption’ would also affect the binding of non-speech stimuli (video clips of object action or musical notes). The auditory and visual stimuli were presented at a range of stimulus onset asynchronies (SOAs) using the method of constant stimuli. Participants made unspeeded temporal order judgments (TOJs) regarding which modality stream had been presented first. The auditory and visual musical and object action stimuli were either matched (e.g., the sight of a note being played on a piano together with the corresponding sound) or else mismatched (e.g., the sight of a note being played on a piano together with the sound of a guitar string being plucked). However, in contrast to the results of Vatakis and Spence’s recent speech study, no significant difference in the accuracy of temporal discrimination performance for the matched versus mismatched video clips was observed. Reasons for this discrepancy are discussed.

The spatial constraint in intersensory pairing: No role in temporal ventriloquism.

A sound presented in temporal proximity to a light can alter the perceived temporal occurrence of that light (temporal ventriloquism). The authors explored whether spatial discordance between the sound and light affects this phenomenon. Participants made temporal order judgments about which of 2 lights appeared first, while they heard sounds before the 1st and after the 2nd light. Sensitivity was higher (i.e., a lower just noticeable difference) when the sound-light interval was approximately 100 ms rather than approximately 0 ms. This temporal ventriloquist effect was unaffected by whether sounds came from the same or a different position as the lights, whether the sounds were static or moved, or whether they came from the same or opposite sides of fixation. Yet, discordant sounds interfered with speeded visual discrimination. These results challenge the view that intersensory interactions in general require spatial correspondence between the stimuli.

An event-related potential investigation of the time-course of temporal ventriloquism

Temporal ventriloquism refers to the phenomenon that a sound presented in close temporal proximity of a visual stimulus attracts its perceived temporal occurrence. Here, we investigate the time-course of the neuronal processes underlying temporal ventriloquism, using event-related brain potentials. To measure shifts in perceived temporal visual occurrence, we used a paradigm in which a sound modulates the magnitude of a visual illusion called the flash-lag effect. A sound presented before the flash reduced both the size of the flash-lag effect and the amplitude of visual N1 compared with when the sound lagged the flash. We attribute the modulation of the flash-lag effect to a modulation of facilitation of visual processing. The time-course (190 ms) and localization (occipitoparietal cortex) of this particular auditory-visual interaction confirms the sensory nature of temporal ventriloquism.

Temporal ventriloquism: sound modulates the flash-lag effect.

A sound presented in close temporal proximity to a visual stimulus can alter the perceived temporal dimensions of the visual stimulus (temporal ventriloquism). In this article, the authors demonstrate temporal ventriloquism in the flash-lag effect (FLE), a visual illusion in which a flash appears to lag relative to a moving object. In Experiment 1, the magnitude and the variability of the FLE were reduced, relative to a silent condition, when a noise burst was synchronized with the flash. In Experiment 2, the sound was presented before, at, or after the flash (+/- approximately 100 ms), and the size of the FLE varied linearly with the delay of the sound. These findings demonstrate that an isolated sound can sharpen the temporal boundaries of a flash and attract its temporal occurrence.

Temporal ventriloquism: crossmodal interaction on the time dimension: 2. Evidence from sensorimotor synchronization

In two experiments, we measured audio-visual crossmodal attraction on the time dimension, using a sensorimotor synchronization task. Synchronization performance made it possible to split up the total crossmodal attraction (demonstrated in earlier studies through inter-modal temporal order judgments) into its modality-specific components, the auditory bias of the visual event's perceived time of occurrence and the visual bias of the auditory event's perceived time of occurrence. Participants were asked to produce tapping movements in synchrony with a sequence of isochronously repeated pacing signals. In Experiment 1, pacing signals were light flashes, each preceded or followed, at one of several stimulus onset asynchronies (SOAs), by an auditory distracter that the participant was instructed to ignore. The timing of the tap was, in spite of that instruction, strongly biased toward the distracter. In Experiment 2, the converse task was used. The pacing signals were auditory and the to be ignored distracters, light flashes. The timing of the taps was biased significantly here also toward the distracter, but to a much lesser extent. Taken together, these results clearly demonstrate that audition plays a bigger role than vision in temporal ventriloquism and is probably generally superior to vision for processing the temporal dimension of events.

Auditory capture of vision: examining temporal ventriloquism

Four experiments investigated whether irrelevant sounds can influence the perception of lights in a visual temporal order judgment task, where participants judged which of two lights appeared first. In Experiment 1, presenting a sound before the first light and after the second light improved performance relative to baseline (sounds appearing simultaneously with the lights), as if the sounds pulled the perception of lights further apart in time. Experiment 2 ruled out an alerting explanation for this effect and indicated that the performance improvement resulted from the second sound trailing the second light. Experiment 3 excluded the possibility that leading or simultaneous sounds were interfering with performance and revealed that only the second sound had an effect within the temporal window known to support multisensory integration. Experiment 4 demonstrated that sounds intervening between the two lights led to a decline in performance, as if the sounds pulled the lights closer together. The results suggest a ‘temporal ventriloquism’ phenomenon analogous to spatial ventriloquism.