Auditory Apparent Motion Research Articles

Saccades create similar mislocalizations in visual and auditory space.

Orienting our eyes to a light, a sound, or a touch occurs effortlessly, despite the fact that sound and touch have to be converted from head- and body-based coordinates to eye-based coordinates to do so. We asked whether the oculomotor representation is also used for localization of sounds even when there is no saccade to the sound source. To address this, we examined whether saccades introduced similar errors of localization judgments for both visual and auditory stimuli. Sixteen subjects indicated the direction of a visual or auditory apparent motion seen or heard between two targets presented either during fixation or straddling a saccade. Compared with the fixation baseline, saccades introduced errors in direction judgments for both visual and auditory stimuli: in both cases, apparent motion judgments were biased in direction of the saccade. These saccade-induced effects across modalities give rise to the possibility of shared, cross-modal location coding for perception and action.

Spatial references and audio-tactile interaction in cross-modal dynamic capture.

In audiotactile dynamic capture, judgment of the direction of an apparent motion stream (such as auditory motion) was impeded (hence 'captured') by the presentation of a concurrent, but directionally opposite apparent motion stream (such as tactile motion) from a distractor modality, leading to a cross-modal dynamic capture (CDC) effect. That is to say, the percentage of correct reporting of the direction of the target motion was reduced. Previous studies have revealed the effect of stimulus onset asynchronies (SOAs) and the potential spatial remapping (by adopting a cross-hands posture) in CDC. However, further exploration of the dynamic capture process under different postures was not available due to the fact that only two levels of time asynchronies were employed (either synchronous or with an SOA of 500 ms). This study introduced a broad range of SOAs (-400 ms to 400 ms, tactile stream preceded auditory stream or vice versa) to explore the time course of audio-tactile interaction in CDC with two spatial references--arms-uncrossed or arms-crossed postures. Participants judged the direction of auditory apparent motion with tactile distractors. The results showed that in the arms-uncrossed condition, the CDC effect was prominent when the auditory-tactile events were in the temporal integration window (0-60 ms). However, with a preceding tactile cueing effect of SOA equal to and above 150 ms, the CDC effect was reduced, and no CDC effect was observed with the arms-crossed posture. These results suggest CDC effect is modulated by both cross-modal interaction and the spatial reference (especially for the distractors). The magnitude of the CDC effects in audiotactile interaction may be accounted for by reliability of tactile spatial-temporal information.

The role of spatiotemporal and spectral cues in segregating short sound events: evidence from auditory Ternus display

Previous studies using auditory sequences with rapid repetition of tones revealed that spatiotemporal cues and spectral cues are important cues used to fuse or segregate sound streams. However, the perceptual grouping was partially driven by the cognitive processing of the periodicity cues of the long sequence. Here, we investigate whether perceptual groupings (spatiotemporal grouping vs. frequency grouping) could also be applicable to short auditory sequences, where auditory perceptual organization is mainly subserved by lower levels of perceptual processing. To find the answer to that question, we conducted two experiments using an auditory Ternus display. The display was composed of three speakers (A, B and C), with each speaker consecutively emitting one sound consisting of two frames (AB and BC). Experiment 1 manipulated both spatial and temporal factors. We implemented three 'within-frame intervals' (WFIs, or intervals between A and B, and between B and C), seven 'inter-frame intervals' (IFIs, or intervals between AB and BC) and two different speaker layouts (inter-distance of speakers: near or far). Experiment 2 manipulated the differentiations of frequencies between two auditory frames, in addition to the spatiotemporal cues as in Experiment 1. Listeners were required to make two alternative forced choices (2AFC) to report the perception of a given Ternus display: element motion (auditory apparent motion from sound A to B to C) or group motion (auditory apparent motion from sound 'AB' to 'BC'). The results indicate that the perceptual grouping of short auditory sequences (materialized by the perceptual decisions of the auditory Ternus display) was modulated by temporal and spectral cues, with the latter contributing more to segregating auditory events. Spatial layout plays a less role in perceptual organization. These results could be accounted for by the 'peripheral channeling' theory.

Mutual Influences of Intermodal Visual/Tactile Apparent Motion and Auditory Motion with Uncrossed and Crossed Arms

Intra-modal apparent motion has been shown to be affected or 'captured' by information from another, task-irrelevant modality, as shown in cross-modal dynamic capture effect. Here we created inter-modal apparent motion between visual and tactile stimuli and investigated whether there are mutual influences between auditory apparent motion and inter-modal visual/tactile apparent motion. Moreover, we examined whether and how the spatial remapping between somatotopic and external reference frames of tactile events affect the cross-modal capture between auditory apparent motion and inter-modal visual/tactile apparent motion, by introducing two arm postures: arms-uncrossed and arms-crossed. In Experiment 1, we used auditory stimuli (auditory apparent motion) as distractors and inter-modal visual/tactile stimuli (inter-modal apparent motion) as targets while in Experiment 2 we reversed the distractors and targets. In Experiment 1, we found a general detrimental influence of arms-crossed posture in the task of discrimination of direction in visual/tactile stream, but in Experiment 2, the influence of arms-uncrossed posture played a significant role in modulating the inter-modal visual/tactile stimuli capturing over auditory apparent motion. In both Experiments, the synchronously presented motion streams led to noticeable directional congruency effect in judging the target motion. Among the different modality combinations, tactile to tactile apparent motion (TT) and visual to visual apparent motion (VV) are two signatures revealing the asymmetric congruency effects. When the auditory stimuli were targets, the congruency effect was largest with VV distractors, lowest with TT distractors; the pattern was reversed when the auditory stimuli were distractors. In addition, across both experiments the congruency effect in visual to tactile (VT) and tactile to visual (TV) apparent motion was intermediate between the effect-sizes in VV and TT. We replicated the above findings with a block-wise design (Experiment 3). In Experiment 4, we introduced static distractor events (visual or tactile stimulus), and found the modulation of spatial remapping of distractors upon AA motion is reduced. These findings suggest that there are mutual but a robust asymmetric influence between intra-modal auditory apparent motion and intermodal visual/tactile apparent motion. We proposed that relative reliabilities in directional information between distractor and target streams, summed over a remapping process between two spatial reference frames, determined this asymmetric influence.

Investigating the In-Between: Multisensory Integration of Auditory and Visual Motion Streams

We investigated audiovisual interactions in motion perception by behavioral experiments testing both, the influence of visual stimuli on auditory apparent motion and the influence of auditory stimuli on visual apparent motion perception. A set of loudspeakers with an LED mounted in the middle of each speaker cone was arranged in a semicircle. Apparent motion streams were presented for each modality alone in the unimodal conditions. In the bimodal conditions, stimuli of the second modality were added to fill the temporal and spatial gaps of the sampled trajectory of the reference stream. The participants' task was to observe the quasi-naturalistic stimulus sequences and to perform a standard classification. The addition of stimuli of the second modality indeed facilitated apparent motion perception. Bimodal presentation increased the upper temporal interval up to which the stimuli could be separated in time while still being perceived as continuous motion. We interpret these results as evidence for an ecologically advantageous audiovisual motion integration mechanism which operates beyond the constraints of strict spatiotemporal coincidence. Functional considerations suggest that this mechanism may represent an amodal stage suited for the processing of both unimodal and bimodal signals.

Auditory Motion Capturing Ambiguous Visual Motion

In this study, it is demonstrated that moving sounds have an effect on the direction in which one sees visual stimuli move. During the main experiment sounds were presented consecutively at four speaker locations inducing left or rightward auditory apparent motion. On the path of auditory apparent motion, visual apparent motion stimuli were presented with a high degree of directional ambiguity. The main outcome of this experiment is that our participants perceived visual apparent motion stimuli that were ambiguous (equally likely to be perceived as moving left or rightward) more often as moving in the same direction than in the opposite direction of auditory apparent motion. During the control experiment we replicated this finding and found no effect of sound motion direction on eye movements. This indicates that auditory motion can capture our visual motion percept when visual motion direction is insufficiently determinate without affecting eye movements.

Visual Motion Interferes with Tactile Motion Perception

Previous studies have demonstrated that visual apparent motion can alter the judgment of auditory apparent motion. We investigated the effect of visual apparent motion on judgments of the direction of tactile apparent motion. When visual motion was presented at the same time as, but in a direction opposite to, tactile motion, accuracy in judging the direction of tactile apparent motion was substantially reduced. This reduction in performance is referred to as 'the congruency effect'. Similar effects were observed when the visual display was placed either near to the tactile display or at some distance from the tactile display (experiment 1). In experiment 2, the relative alignment between the visual and tactile directions of motion was varied. The size of the congruency effect was similar at 0 degrees and 45 degrees alignments but much reduced at a 90 degrees alignment. In experiment 3, subjects made confidence ratings of their judgments of the direction of the tactile motion. The results indicated that the congruency effect was not due to subjects being unsure of the direction of motion and being forced to guess. In experiment 4, static visual stimuli were shown to have no effect on the judgments of direction of the tactile stimuli. The extent to which the congruency effect reflects capture effects and is the result of perceptual versus post-perceptual processes is discussed.

Perceptual and decisional contributions to audiovisual interactions in the perception of apparent motion: A signal detection study

Motion information available to different sensory modalities can interact at both perceptual and post-perceptual (i.e., decisional) stages of processing. However, to date, researchers have only been able to demonstrate the influence of one of these components at any given time, hence the relationship between them remains uncertain. We addressed the interplay between the perceptual and post-perceptual components of information processing by assessing their influence on performance within the same experimental paradigm. We used signal detection theory to discriminate changes in perceptual sensitivity (d′) from shifts in response criterion (c) in performance on a detection (Experiment 1) and a classification (Experiment 2) task regarding the direction of auditory apparent motion streams presented in noise. In the critical conditions, a visual motion distractor moving either leftward or rightward was presented together with the auditory motion. The results demonstrated a significant decrease in sensitivity to the direction of the auditory targets in the crossmodal conditions as compared to the unimodal baseline conditions that was independent of the relative direction of the visual distractor. In addition, we also observed significant shifts in response criterion, which were dependent on the relative direction of the distractor apparent motion. These results support the view that the perceptual and decisional components involved in audiovisual interactions in motion processing can coexist but are largely independent of one another.

Assessing the effect of visual and tactile distractors on the perception of auditory apparent motion

In this study we investigated the effect of the directional congruency of tactile, visual, or bimodal visuotactile apparent motion distractors on the perception of auditory apparent motion. Participants had to judge the direction in which an auditory apparent motion stream moved (left-to-right or right-to-left) while trying to ignore one of a range of distractor stimuli, including unimodal tactile or visual, bimodal visuotactile, and crossmodal (i.e., composed of one visual and one tactile stimulus) distractors. Significant crossmodal dynamic capture effects (i.e., better performance when the target and distractor stimuli moved in the same direction rather than in opposite directions) were demonstrated in all conditions. Bimodal distractors elicited more crossmodal dynamic capture than unimodal distractors, thus providing the first empirical demonstration of the effect of information presented simultaneously in two irrelevant sensory modalities on the perception of motion in a third (target) sensory modality. The results of a second experiment demonstrated that the capture effect reported in the crossmodal distractor condition was most probably attributable to the combined effect of the individual static distractors (i.e., to ventriloquism) rather than to any emergent property of crossmodal apparent motion.

Assessing automaticity in the audiovisual integration of motion

This study examined the multisensory integration of visual and auditory motion information using a methodology designed to single out perceptual integration processes from post-perceptual influences. We assessed the threshold stimulus onset asynchrony (SOA) at which the relative directions (same vs. different) of simultaneously presented visual and auditory apparent motion streams could no longer be discriminated (Experiment 1). This threshold was higher than the upper threshold for direction discrimination (left vs. right) of each individual modality when presented in isolation (Experiment 2). The poorer performance observed in bimodal displays was interpreted as a consequence of automatic multisensory integration of motion information. Experiment 3 supported this interpretation by ruling out task differences as the explanation for the higher threshold in Experiment 1. Together these data provide empirical support for the view that multisensory integration of motion signals can occur at a perceptual level.

A Comparison of Auditory and Visual Apparent Motion Presented Individually and with Crossmodal Moving Distractors

Unimodal auditory and visual apparent motion (AM) and bimodal audiovisual AM were investigated to determine the effects of crossmodal integration on motion perception and direction-of-motion discrimination in each modality. To determine the optimal stimulus onset asynchrony (SOA) ranges for motion perception and direction discrimination, we initially measured unimodal visual and auditory AMs using one of four durations (50, 100, 200, or 400 ms) and ten SOAs (40-450 ms). In the bimodal conditions, auditory and visual AM were measured in the presence of temporally synchronous, spatially displaced distractors that were either congruent (moving in the same direction) or conflicting (moving in the opposite direction) with respect to target motion. Participants reported whether continuous motion was perceived and its direction. With unimodal auditory and visual AM, motion perception was affected differently by stimulus duration and SOA in the two modalities, while the opposite was observed for direction of motion. In the bimodal audiovisual AM condition, discriminating the direction of motion was affected only in the case of an auditory target. The perceived direction of auditory but not visual AM was reduced to chance levels when the crossmodal distractor direction was conflicting. Conversely, motion perception was unaffected by the distractor direction and, in some cases, the mere presence of a distractor facilitated movement perception.

When does visual perceptual grouping affect multisensory integration?

Several studies have shown that the direction in which a visual apparent motion stream moves can influence the perceived direction of an auditory apparent motion stream (an effect known as cross-modal dynamic capture). However, little is known about the role that intramodal perceptual grouping processes play in the multisensory integration of motion information. The present study was designed to investigate the time course of any modulation of the cross-modal dynamic capture effect by the nature of the perceptual grouping taking place within vision. Participants were required to judge the direction of an auditory apparent motion stream while trying to ignore visual apparent motion streams presented in a variety of different configurations. Our results demonstrate that the cross-modal dynamic capture effect was influenced more by visual perceptual grouping when the conditions for intramodal perceptual grouping were set up prior to the presentation of the audiovisual apparent motion stimuli. However, no such modulation occurred when the visual perceptual grouping manipulation was established at the same time as or after the presentation of the audiovisual stimuli. These results highlight the importance of the unimodal perceptual organization of sensory information to the manifestation of multisensory integration.

Neural Processing of Auditory Looming in the Human Brain

Acoustic intensity change, along with interaural, spectral, and reverberation information, is an important cue for the perception of auditory motion [1–4]. Approaching sound sources produce increases in intensity, and receding sound sources produce corresponding decreases. Human listeners typically overestimate increasing compared to equivalent decreasing sound intensity [5, 6] and underestimate the time to contact of approaching sound sources [2, 7]. These characteristics could provide a selective advantage by increasing the margin of safety for response to looming objects. Here, we used dynamic intensity and functional magnetic resonance imaging to examine the neural underpinnings of the perceptual priority for rising intensity. We found that, consistent with activation by horizontal and vertical auditory apparent motion paradigms [8–10], rising and falling intensity activated the right temporal plane more than constant intensity. Rising compared to falling intensity activated a distributed neural network subserving space recognition, auditory motion perception, and attention [8–23] and comprising the superior temporal sulci and the middle temporal gyri, the right temporoparietal junction, the right motor and premotor cortices, the left cerebellar cortex, and a circumscribed region in the midbrain. This anisotropic processing of acoustic intensity change may reflect the salience of rising intensity produced by looming sources in natural environments.

The ventriloquist in motion: Illusory capture of dynamic information across sensory modalities

Integrating dynamic information across the senses is crucial to survival. However, most laboratory studies have only examined sensory integration for static events. Here we demonstrate that strong crossmodal integration can also occur for an emergent attribute of dynamic arrays, specifically the direction of apparent motion. The results of the present study show that the perceived direction of auditory apparent motion is strongly modulated by apparent motion in vision, and that both spatial and temporal factors play a significant role in this crossmodal effect. We also demonstrate that a split-brain patient who does not perceive visual apparent motion across the midline is immune to this audiovisual dynamic capture effect, highlighting the importance of motion being experienced in order for this new multisensory illusion to occur.

Motion processing in the auditory cortex of the rufous horseshoe bat: role of GABAergic inhibition.

This study examined the influence of inhibition on motion-direction-sensitive responses of neurons in the dorsal fields of auditory cortex of the rufous horseshoe bat. Responses to auditory apparent motion stimuli were recorded extracellularly from neurons while microiontophoretically applying gamma-aminobutyric acid (GABA) and the GABAA receptor antagonist bicuculline methiodide (BMI). Neurons could respond with a directional preference exhibiting stronger responses to one direction of motion or a shift of receptive field (RF) borders depending on direction of motion. BMI influenced the motion direction sensitivity of 53% of neurons. In 21% of neurons the motion-direction sensitivity was decreased by BMI by decreasing either directional preference or RF shift. In neurons with a directional preference, BMI increased the spike number for the preferred direction by a similar amount as for the nonpreferred direction. Thus, inhibition was not direction specific. BMI increased motion-direction sensitivity by either increasing directional preference or magnitude of RF shifts in 22% of neurons. Ten percent of neurons changed their response from a RF shift to a directional preference under BMI. In these neurons, the observed effects could often be better explained by adaptation of excitation rather than inhibition. The results suggest, that adaptation of excitation, as well as cortex specific GABAergic inhibition, contribute to motion-direction sensitivity in the auditory cortex of the rufous horseshoe bat.

Spatial receptive fields of inferior colliculus neurons to auditory apparent motion in free field.

We examined responses from 91 single-neurons in the inferior colliculus (IC) of anesthetized guinea pigs to auditory apparent motion in the free field. Apparent motion was generated by presenting 100-ms tone bursts, separated by 50-ms silent intervals, at consecutive speaker positions in an array of 11 speakers, positioned in an arc +/-112.5 degrees around midline. Most neurons demonstrated discrete spatial receptive fields (SRFs) to apparent motion in the clockwise and anti-clockwise directions. However, SRFs showed marked differences for apparent motion in opposite directions. In virtually all neurons, mean best azimuthal positions for SRFs to opposite directions occurred at earlier positions in the motion sweep, producing receptive fields to the two directions of motion that only partially overlapped. Despite this, overall spike counts to the two directions were similar for equivalent angular velocities. Responses of 28 neurons were recorded to stimuli with different duration silent intervals between speaker presentations, mimicking different apparent angular velocities. Increasing the stimulus OFF time increased neuronal discharge rates, particularly at later portions of the apparent motion sweep, and reduced the differences in the SRFs to opposite motion directions. Consequently SRFs to both directions broadened and converged with decreasing motion velocity. This expansion was most obvious on the outgoing side of the each SRF. Responses of 11 neurons were recorded to short (90 degrees ) partially overlapping apparent motion sweeps centered at different spatial positions. Nonoverlapping response profiles were recorded in 9 of the 11 neurons tested and confirmed that responses at each speaker position were dependent on the preceding response history. Together these data are consistent with the suggestion that a mechanism of adaptation of excitation contributes to the apparent sensitivity of IC neurons to auditory motion cues. In addition, the data indicate that the sequential activation of an array of speakers to produce apparent auditory motion may not be an optimal stimulus paradigm to separate the temporal and spatial aspects of auditory motion processing.

Marking the starting and ending points of auditory apparent motion: Effect of timing, spatial separation, and azimuth

Participants identified the path of auditory apparent motion (AAM), produced by the appropriate timing (burst duration and stimulus onset asynchrony, SOA) of two successive noise bursts (high-pass A-weighted noise at 55 dB). Three burst duration/SOA combinations were tested: 100/50 ms, 400/100 ms, and 400/600 ms. The first two combinations produce AAM; the last was a nonmoving control. The stimulus sequence was presented through two loudspeakers selected from a horizontal array so as to produce two spatial separations (16 and 33°) and three midpoint locations (−23, 0, and 23° azimuth). The actual starting and ending locations were jittered around each midpoint. Participants faced a large cloth screen having a grid pattern and were instructed to identify the cells corresponding to the starting and ending points of the stimulus sequence. At timing values producing motion, the path of motion was displaced in the direction of the motion with the greatest direction effect occurring at 100/50 ms. However, the midpoint between the loudspeakers also affected the path of AAM. At −22° azimuth, the path was displaced toward the left, regardless of motion direction. At 22° azimuth the path was displaced toward the right. This midpoint effect was obtained only at timing values producing AAM.

The effect of timing and spatial separation on the velocity of auditory apparent motion.

Previously, it was shown that the minimum conditions for the illusion of auditory apparent motion (AAM) depend on stimulus timing but not spatial separation. In the present experiment, the effects of stimulus timing and source separation on the perceived velocity of AAM were examined. Eight listeners estimated the velocity, duration, and distance traveled of AAM, using a no-modulus, magnitude estimation procedure. Four burst durations (25, 50, 100, and 300 msec), 10 stimulus onset asynchronies (SOAs; 30, 40, 50, 60, 70, 80, 90, 100, 110, and 120 msec) and two separations (10 degrees and 40 degrees) were tested. Perceived velocity estimates were related to the total duration (burst duration + SOA) of the stimulus sequence. The effect of separation on velocity was extremely small but statistically significant. These results are similar to those obtained previously on the minimum conditions for AAM. Duration estimates were related only to total duration, but separation estimates were related to both separation and total duration. These results suggest that velocity is possibly a primary dimension of AAM that is independent of source separation.

Auditory apparent motion between sine waves differing in frequency.

The effects of frequency differences between the lead and lag stimuli on auditory apparent motion (AAM--the perception of continuous changes in the location of a sound image over time) were examined in two experiments. In experiment 1, three standard frequencies (500, 1000, and 5000 Hz) and three SOAs (40, 60, and 100 ms) were tested. Both standard frequency and stimulus onset asynchrony (SOA) were constant throughout a session. Eleven comparison frequencies were tested within each session, with the range dependent on the standard frequency. At standard frequencies of 500 and 1000 Hz, AAM was heard when the frequencies of the lead and lag stimuli were within 100 Hz of each other. At 5000 Hz, the range of frequencies producing AAM increased with SOA. In experiment 2, two standards (500 and 5000 Hz) were tested with a wider range of SOAs (10-210 ms) varied within a session, and a narrower range of comparison frequencies. Here, comparison frequency was constant throughout a session. At 500 Hz, the SOAs producing AAM did not depend on comparison frequency. At 5000 Hz, the SOAs producing AAM increased with comparison frequency, consistent with Korte's third law of visual apparent motion.

The effect of burst duration, interstimulus onset interval, and loudspeaker arrangement on auditory apparent motion in the free field

The illusion of auditory apparent motion (AAM) was examined in order to determine the burst durations and interstimulus onset intervals (ISOIs) at which AAM is heard when spatial information regarding source location was varied. In the first experiment AAM was examined in the free field under monaural and binaural listening conditions. AAM was heard at the same burst duration-ISOI combinations for both listening conditions, but the location of the lead source could be determined only under binaural listening. In the second experiment AAM was measured with two and three sound sources. The number of sources did not affect the burst duration-ISOI combinations that produced AAM, but did affect the determination of the location of the lead source. In the third experiment AAM was tested when the sources were located in the median plane. The sources were located either at 0 degrees and 180 degrees azimuth, or both at 0 degrees azimuth, one in the horizontal plane and one 20 degrees above. The location of the speakers did not affect the timing requirements for the perception of AAM, only the timing requirements for the detection of the lead source. In the fourth experiment, AAM was measured when the vertical separation between the sources was either 2.5 degrees or 20 degrees. AAM was heard at both separations, even though 2.5 degrees is less than the vertical MAA. In each of these experiments only burst duration and ISOI determined whether motion was heard. Localization cues were important only for the determination of the direction of motion.