Temporal Ventriloquism Research Articles

Sound improves diminished visual temporal sensitivity in schizophrenia

Visual temporal processing and multisensory integration (MSI) of sound and vision were examined in individuals with schizophrenia using a visual temporal order judgment (TOJ) task. Compared to a non-psychiatric control group, persons with schizophrenia were less sensitive judging the temporal order of two successively presented visual stimuli. However, their sensitivity to visual temporal order improved as in the control group when two accessory sounds were added (temporal ventriloquism). These findings indicate that individuals with schizophrenia have diminished sensitivity to visual temporal order, but no deficits in the integration of low-level auditory and visual information.

Quantifying temporal ventriloquism in audiovisual synchrony perception

The integration of visual and auditory inputs in the human brain works properly only if the components are perceived in close temporal proximity. In the present study, we quantified cross-modal interactions in the human brain for audiovisual stimuli with temporal asynchronies, using a paradigm from rhythm perception. In this method, participants had to align the temporal position of a target in a rhythmic sequence of four markers. In the first experiment, target and markers consisted of a visual flash or an auditory noise burst, and all four combinations of target and marker modalities were tested. In the same-modality conditions, no temporal biases and a high precision of the adjusted temporal position of the target were observed. In the different-modality conditions, we found a systematic temporal bias of 25-30 ms. In the second part of the first and in a second experiment, we tested conditions in which audiovisual markers with different stimulus onset asynchronies (SOAs) between the two components and a visual target were used to quantify temporal ventriloquism. The adjusted target positions varied by up to about 50 ms and depended in a systematic way on the SOA and its proximity to the point of subjective synchrony. These data allowed testing different quantitative models. The most satisfying model, based on work by Maij, Brenner, and Smeets (Journal of Neurophysiology 102, 490-495, 2009), linked temporal ventriloquism and the percept of synchrony and was capable of adequately describing the results from the present study, as well as those of some earlier experiments.

Intersensory binding across space and time: A tutorial review

Spatial ventriloquism refers to the phenomenon that a visual stimulus such as a flash can attract the perceived location of a spatially discordant but temporally synchronous sound. An analogous example of mutual attraction between audition and vision has been found in the temporal domain, where temporal aspects of a visual event, such as its onset, frequency, or duration, can be biased by a slightly asynchronous sound. In this review, we examine various manifestations of spatial and temporal attraction between the senses (both direct effects and aftereffects), and we discuss important constraints on the occurrence of these effects. Factors that potentially modulate ventriloquism-such as attention, synesthetic correspondence, and other cognitive factors-are described. We trace theories and models of spatial and temporal ventriloquism, from the traditional unity assumption and modality appropriateness hypothesis to more recent Bayesian and neural network approaches. Finally, we summarize recent evidence probing the underlying neural mechanisms of spatial and temporal ventriloquism.

Direction of visual apparent motion driven by perceptual organization of cross-modal signals

A critical function of the human brain is to determine the relationship between sensory signals. In the case of signals originating from different sensory modalities, such as audition and vision, several processes have been proposed that may facilitate perception of correspondence between two signals despite any temporal discrepancies in physical or neural transmission. One proposal, temporal ventriloquism, suggests that audio-visual temporal discrepancies can be resolved with a capture of visual event timing by that of nearby auditory events. Such an account implies a fundamental change in the timing representations of the involved events. Here we examine if such changes are necessary to account for a recently demonstrated effect, the modulation of visual apparent motion direction by audition. By contrast, we propose that the effect is driven by segmentation of the visual sequence on the basis of perceptual organization in the cross-modal sequence. Using different sequences of cross-modal (auditory and tactile) events, we found that the direction of visual apparent motion was not consistent with a temporal capture explanation. Rather, reports of visual apparent motion direction were dictated by perceptual organization within cross-modal sequences, determined on the basis of apparent relatedness. This result adds to the growing literature indicating the importance of apparent relatedness and sequence segmentation in apparent timing. Moreover, it demonstrates that, contrary to previous findings, cross-modal interaction can play a critical role in determining organization of signals within a single sensory modality.

Multimodal Integration of Interval Duration: Temporal Ventriloquism or Changes in Pacemaker Rate?

Previous studies demonstrated that perceived duration of visual intervals is strongly influenced by conflicting auditory intervals. However, it remains unclear which mechanisms underlie this multimodal integration of interval duration. To investigate this issue, we employed a reproduction task with empty (Experiment 1) or filled (Experiment 2) intervals, and a paired-comparison task (Experiment 3) to assess perceived duration of visually marked intervals, which could be accompanied by auditory marked intervals with congruent (same) or conflicting (longer and shorter) durations. First, we predicted that conflicting auditory intervals would bias the perceived duration of visual intervals towards the duration of the auditory ones. Second, according to pacemaker-accumulator models, two different mechanisms might contribute to multimodal interval integration: changes in pacemaker rate or changes of the switch component based on temporal ventriloquism effects. In the former case, the multimodal bias effect should increase with increasing interval duration. In the latter case, the effect should remain constant across interval durations. All experiments showed a strong influence of auditory interval duration on perceived visual duration. In Experiments 1 and 2, this bias effect unexpectedly decreased with increasing interval duration, which might be due to limitations of the employed duration reproduction method. In Experiment 3, however, the observed multimodal bias effect remained clearly constant across all interval durations. This finding supports the idea that multimodal integration of conflicting time intervals is mainly determined by a temporal ventriloquism effect, which affects the switch component of a pacemaker-accumulator mechanism. In addition, we demonstrated that bimodal congruent intervals are perceived as longer than unimodal visual ones. For filled (Experiment 2), but not for empty (Experiment 1) intervals, this effect seems to be caused by an increase in pacemaker rate.

Static sound timing alters sensitivity to low-level visual motion

Visual motion processing is essential to survival in a dynamic world and is probably the best-studied facet of visual perception. It has been recently discovered that the timing of brief static sounds can bias visual motion perception, an effect attributed to "temporal ventriloquism" whereby the timing of the sounds "captures" the timing of the visual events. To determine whether this cross-modal interaction is dependent on the involvement of higher-order attentive tracking mechanisms, we used near-threshold motion stimuli that isolated low-level pre-attentive visual motion processing. We found that the timing of brief sounds altered sensitivity to these visual motion stimuli in a manner that paralleled changes in the timing of the visual stimuli. Our findings indicate that auditory timing impacts visual motion processing very early in the processing hierarchy and without the involvement of higher-order attentional and/or position tracking mechanisms.

Changes in visual apparent motion direction by cross-modal interaction are not dependent on temporal ventriloquism

Changes in visual apparent motion direction by cross-modal interaction are not dependent on temporal ventriloquism

Dissociating Attention and Audiovisual Integration in the Sound-Facilitatory Effect on Metacontrast Masking

In metacontrast masking, target visibility is impaired by a subsequent non-overlapping contour-matched mask, a phenomenon attributed to low-level processing. Previously we found that sound could reduce metacontrast masking (Yeh & Chen, 2010), and yet how it exerts its effect and whether the sound-triggered attention system plays a major role remains unsolved. Here we examine whether the sound-facilitatory effect is caused by alertness, attentional cueing, or audiovisual integration. Two sounds were either presented simultaneously with the target and the mask respectively, or one preceded the target by 100 ms and the other followed the mask 100 ms afterwards. No-sound and one-sound conditions were used for comparison. Participants discriminated the truncated part (up or down) of the target, with four target-to-mask SOAs (14ms, 43ms, 114ms, and 157ms) mixedly presented. Results showed that the attentional cueing effect was evident when compared to the one-leading sound condition. Additionally, selective (rather than overall) improvement in accuracy and RT as a function of SOA was found with synchronized sound than without, suggesting audiovisual integration but not alertness. The audio-visual integration effect is attributed to enhanced temporal resolution but not temporal ventriloquism.

Sound affects the speed of visual processing.

The authors examined the effects of a task-irrelevant sound on visual processing. Participants were presented with revolving clocks at or around central fixation and reported the hand position of a target clock at the time an exogenous cue (1 clock turning red) or an endogenous cue (a line pointing toward 1 of the clocks) was presented. A spatially irrelevant sound presented 100 ms before the cue speeded visual latency when compared with a sound presented 100 ms after the cue. The effect of the sound was larger the farther the target was from fixation, and it was larger for endogenous than exogenous cues. A visual temporal warning signal had different effects on perceptual latency. These results demonstrate that an asynchronous sound can shift the perceived time of occurrence of a visual cue (temporal ventriloquism) and speed the velocity of the attentional shift toward the target. Sounds thus have multiple effects on visual perception.

Capture of Intermodal Visual/Tactile Apparent Motion by Moving and Static Sound

Apparent motion can occur within a particular modality or between modalities, in which a visual or tactile stimulus at one location is perceived as moving towards the location of the subsequent tactile or visual stimulus. Intramodal apparent motion has been shown to be affected or 'captured' by information from another, task-irrelevant modality, as in spatial or temporal ventriloquism. Here we investigate whether and how intermodal apparent motion is affected by motion direction cues or temporal interval information from a third modality. We demonstrated that both moving and asynchronous static sounds can capture intermodal (visual-tactile and tactile-visual) apparent motion; moreover, while the auditory direction cues have less impact upon the perception of intramodal visual apparent motion than upon the perception of intramodal tactile or intermodal visual/tactile apparent motion, the auditory temporal information has equivalent impacts upon both intramodal and intermodal apparent motion. These findings suggest intermodal apparent motion is susceptible to the influence of dynamic or static auditory information in similar ways as intramodal visual or tactile apparent motion.

Temporal ventriloquism in a purely temporal context.

This study examines how audiovisual signals are combined in time for a temporal analogue of the ventriloquist effect in a purely temporal context, that is, no spatial grounding of signals or other spatial facilitation. Observers were presented with two successive intervals, each defined by a 1250-ms tone, and indicated in which interval a brief audiovisual stimulus (visual flash + noise burst) occurred later. In "test" intervals, the audiovisual stimulus was presented with a small asynchrony, while in "probe" intervals it was synchronous and presented at various times guided by an adaptive staircase to find the perceived temporal location of the asynchronous stimulus. As in spatial ventriloquism, and consistent with maximum likelihood estimation (MLE), the asynchronous audiovisual signal was shifted toward the more reliably localized component (audition, for all observers). Moreover, these temporal shifts could be forward or backward in time, depending on the asynchrony order, suggesting perceived timing is not entirely determined by physical timing. However, the critical signature of MLE combination--better bimodal than unimodal precision--was not found. Regardless of the underlying model, these results demonstrate temporal ventriloquism in a paradigm that is defined in a purely temporal context.

No effect of synesthetic congruency on temporal ventriloquism

A sound presented in temporal proximity to a light can alter the perceived temporal occurrence of that light (temporal ventriloquism). Recent studies have suggested that pitch–size synesthetic congruency (i.e., a natural association between the relative pitch of a sound and the relative size of a visual stimulus) might affect this phenomenon. To reexamine this, participants made temporal order judgements about small- and large-sized visual stimuli while high- or low-pitched tones were presented before the first and after the second light. We replicated a previous study showing that, at large sound–light intervals, sensitivity for visual temporal order was better for synesthetically congruent than for incongruent pairs. However, this congruency effect could not be attributed to temporal ventriloquism, since it disappeared at short sound–light intervals, if compared with a synchronous audiovisual baseline condition that excluded response biases. In addition, synesthetic congruency did not affect temporal ventriloquism even if participants were made explicitly aware of congruency before testing. Our results thus challenge the view that synesthetic congruency affects temporal ventriloquism.

Crossmodal duration perception involves perceptual grouping, temporal ventriloquism, and variable internal clock rates

Here, we investigate how audiovisual context affects perceived event duration with experiments in which observers reported which of two stimuli they perceived as longer. Target events were visual and/or auditory and could be accompanied by nontargets in the other modality. Our results demonstrate that the temporal information conveyed by irrelevant sounds is automatically used when the brain estimates visual durations but that irrelevant visual information does not affect perceived auditory duration (Experiment 1). We further show that auditory influences on subjective visual durations occur only when the temporal characteristics of the stimuli promote perceptual grouping (Experiments 1 and 2). Placed in the context of scalar expectancy theory of time perception, our third and fourth experiments have the implication that audiovisual context can lead both to changes in the rate of an internal clock and to temporal ventriloquism-like effects on perceived on- and offsets. Finally, intramodal grouping of auditory stimuli diminished any crossmodal effects, suggesting a strong preference for intramodal over crossmodal perceptual grouping (Experiment 5).

Auditory temporal modulation of the visual Ternus effect: the influence of time interval

Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed 'temporal ventriloquism effect.' Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: 'element motion' or 'group motion'. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion (Experiments 1 and 3). However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.

Temporal auditory capture does not affect the time course of saccadic mislocalization of visual stimuli

Irrelevant sounds can "capture" visual stimuli to change their apparent timing, a phenomenon sometimes termed "temporal ventriloquism". Here we ask whether this auditory capture can alter the time course of spatial mislocalization of visual stimuli during saccades. We first show that during saccades, sounds affect the apparent timing of visual flashes, even more strongly than during fixation. However, this capture does not affect the dynamics of perisaccadic visual distortions. Sounds presented 50 ms before or after a visual bar (that change perceived timing of the bars by more than 40 ms) had no measurable effect on the time courses of spatial mislocalization of the bars, in four subjects. Control studies showed that with barely visible, low-contrast stimuli, leading, but not trailing, sounds can have a small effect on mislocalization, most likely attributable to attentional effects rather than auditory capture. These findings support previous studies showing that integration of multisensory information occurs at a relatively late stage of sensory processing, after visual representations have undergone the distortions induced by saccades.

Auditory dominance over vision in the perception of interval duration

The "ventriloquist effect" refers to the fact that vision usually dominates hearing in spatial localization, and this has been shown to be consistent with optimal integration of visual and auditory signals (Alais and Burr in Curr Biol 14(3):257-262, 2004). For temporal localization, however, auditory stimuli often "capture" visual stimuli, in what has become known as "temporal ventriloquism". We examined this quantitatively using a bisection task, confirming that sound does tend to dominate the perceived timing of audio-visual stimuli. The dominance was predicted qualitatively by considering the better temporal localization of audition, but the quantitative fit was less than perfect, with more weight being given to audition than predicted from thresholds. As predicted by optimal cue combination, the temporal localization of audio-visual stimuli was better than for either sense alone.

Adaptation to audiovisual asynchrony modulates the speeded detection of sound

The brain adapts to asynchronous audiovisual signals by reducing the subjective temporal lag between them. However, it is currently unclear which sensory signal (visual or auditory) shifts toward the other. According to the idea that the auditory system codes temporal information more precisely than the visual system, one should expect to find some temporal shift of vision toward audition (as in the temporal ventriloquism effect) as a result of adaptation to asynchronous audiovisual signals. Given that visual information gives a more exact estimate of the time of occurrence of distal events than auditory information (due to the fact that the time of arrival of visual information regarding an external event is always closer to the time at which this event occurred), the opposite result could also be expected. Here, we demonstrate that participants' speeded reaction times (RTs) to auditory (but, critically, not visual) stimuli are altered following adaptation to asynchronous audiovisual stimuli. After receiving "baseline" exposure to synchrony, participants were exposed either to auditory-lagging asynchrony (VA group) or to auditory-leading asynchrony (AV group). The results revealed that RTs to sounds became progressively faster (in the VA group) or slower (in the AV group) as participants' exposure to asynchrony increased, thus providing empirical evidence that speeded responses to sounds are influenced by exposure to audiovisual asynchrony.

Audio-Visual Organisation and the Temporal Ventriloquism Effect between Grouped Sequences: Evidence That Unimodal Grouping Precedes Cross-Modal Integration

Cross-modal grouping interactions between auditory and visual sequences have previously been demonstrated (O'Leary and Rhodes 1984, Perception & Psychophysics 33 565-569; Rahne et al 2008, Brain Research 1220 118-131). Three experiments were conducted here to determine whether cross-modal interaction precedes unimodal grouping and whether the temporal ventriloquism effect could be found between grouped auditory and visual sequences. We used a repeating four-tone auditory sequence (high-middle-middle-low) where the middle tones could group with either the high tone or the low tone, paired with a sequence of light flashes with a single flash one side and three the other. Experiment 1 showed that the temporal position of the isolated flash in the visual sequence had no effect on which of the auditory tones were perceived as isolated. Experiments 2 and 3 demonstrated that the temporal ventriloquism effect occurs between grouped auditory and visual sequences, as participants reported that the isolated light and tone from grouped visual and auditory sequences seemed to synchronise when they were between 120 and 240 ms apart. These results suggest that unisensory grouping can occur prior to cross-modal interaction.

Sounds change four-dot masking

The temporal occurrence of a flash can be shifted towards a slightly offset sound (temporal ventriloquism). Here we examined whether four-dot masking is affected by this phenomenon. In Experiment 1, we demonstrate that there is release from four-dot masking if two sounds – one before the target and one after the mask – are presented at ∼100ms intervals rather than at ∼0ms intervals or a silent condition. In Experiment 2, we show that the release from masking originates from an alerting effect of the first sound, and a temporal ventriloquist effect from the first and second sounds that lengthened the perceived interval between target and mask, thereby leaving more time for the target to consolidate. Results thus show that sounds penetrate the visual system at more than one level.

Direction of Visual Apparent Motion Driven Solely by Timing of a Static Sound

In temporal ventriloquism, auditory events can illusorily attract perceived timing of a visual onset [1-3]. We investigated whether timing of a static sound can also influence spatio-temporal processing of visual apparent motion, induced here by visual bars alternating between opposite hemifields. Perceived direction typically depends on the relative interval in timing between visual left-right and right-left flashes (e.g., rightwards motion dominating when left-to-right interflash intervals are shortest [4]). In our new multisensory condition, interflash intervals were equal, but auditory beeps could slightly lag the right flash, yet slightly lead the left flash, or vice versa. This auditory timing strongly influenced perceived visual motion direction, despite providing no spatial auditory motion signal whatsoever. Moreover, prolonged adaptation to such auditorily driven apparent motion produced a robust visual motion aftereffect in the opposite direction, when measured in subsequent silence. Control experiments argued against accounts in terms of possible auditory grouping, or possible attention capture. We suggest that the motion arises because the sounds change perceived visual timing, as we separately confirmed. Our results provide a new demonstration of multisensory influences on sensory-specific perception [5], with timing of a static sound influencing spatio-temporal processing of visual motion direction.