Understanding cross-modal interactions of spatial and temporal information from a cortical perspective

Abstract

In the ventriloquism effect, perception of visual spatial information biases judgments of auditory events, yet there is little effect of auditory stimuli on perception of visual locations. In the flash-beep illusion, the number of auditory “beeps” biases judgments of the count of visual flashes, but visual flashes have little effect on perceived number of auditory events. These asymmetries suggest that vision is the “natural” sensory modality for coding spatial information, while audition is specialized for representing temporal information. Here, we review recent behavioral and neuroimaging evidence from our labs exploring the neural underpinnings of such perceptual asymmetries. Specifically, we find that there are distinct frontal cortical networks associated with visual information and auditory information. Yet these networks can be recruited by the other sensory modality, depending on task demands. For instance, when judging spatial aspects of auditory inputs, neural areas associated with visual processing are recruited; when judging temporal aspects of visual inputs, areas associated with auditory processing are activated. We also find another asymmetry: knowing when a spatial event is going to occur helps listeners judge location, but knowing where an event will occur does not help judgments about that event's timing. These kinds of studies help elucidate how temporal and spatial information is encoded in the brain, and the neural mechanisms by which visual-spatial and auditory-temporal information interact.