Visual capture and changes in response times in multisensory localization tasks

Abstract

In the real-world, environmental objects are often both seen and heard. Visual stimuli can influence the accuracy, variability, and timing of the listener’s responses to auditory targets. One well-known example of this visual influence is the Ventriloquist Illusion or visual capture of the perceived sound source location. However, less is known about how vision affects the timing of sensorimotor output driven by auditory events. We hypothesize that response time may manifest the influences of two different visual factors—stimulus features (e.g., spatial congruence) and environmental features (e.g., contextual reference). We measured eye saccades, a natural orienting behavior, in response to auditory and visual stimuli with different degrees of spatial alignment in two visual environments (with or without contextual reference). Results showed both visual factors can influence the timing of saccades to auditory targets. Response time was longer when auditory, and visual signals are spatially separated. Response time was also longer in the non-referenced than referenced environments, but the effect size was small. These results demonstrate response time as a valuable metric for understanding multisensory perception. Further studies are needed to investigate the patterns of changes in both response time and response accuracy to gain new insights into auditory localization in real-life settings.