Hearing motion in "the mind's ear" - evidence for a vision-to-sound synesthesia

Abstract

Here we report a previously undescribed vision-to-sound synesthesia. We studied three healthy individuals for whom seeing visual flashes or motion causes the perception of sound (e.g. tapping or buzzing). Consistent with other synesthesias, the visually-induced sound percepts are automatic and cannot be turned off. This synesthesia conferred a distinct advantage on a visual psychophysics task. Synesthetes (n=2) and control (n=8) subjects were presented with rapid rhythmic sequences (similar to Morse code) composed of either auditory beeps or of visual flashes. Subjects judged whether two successive sequences (either both auditory or both visual) were the same or different sequence (2-IFC trials). All subjects performed well on auditory trials (syn: 86% correct, controls: 88%). However, only synesthetes performed well on visual trials with controls performing near 50% chance level (syn: 77%, controls: 57%; difference p[[lt]]0.0005). These results are consistent with synesthetes being able to hear the rhythms on visual trials, with the sound percepts being closely temporally linked to the visual inducers. Next, we measured BOLD fMRI responses to viewing of moving vs. stationary visual dots (a standard MT+ localizer stimulus). Synesthetes (n=3) reported sound percepts for the moving but not the stationary condition and showed enhanced responses to visual motion compared to controls (n=8) on the bilateral superior temporal sulcus (STS), a previously reported audiovisual integration site. The same region was highly responsive to visual stimuli in both synesthete and control subjects, when given a task that encouraged temporally-linked sound imagery. Furthermore, the same STS region responded to purely auditory stimuli in both synesthetes and controls, verifying its identity as an audiovisual convergence region. These fMRI results suggest that vision-to-sound synesthetes have an exaggerated form of multisensory integration that occurs within the normal population.