Natural scenes have matched amplitude-modulated sounds that systematically influence visual scanning

Abstract

We have previously demonstrated a linear perceptual relationship between auditory amplitude-modulation (AM) rate and visual spatial-frequency using gabors as the visual stimuli. Can this frequency-based auditory–visual association influence perception of natural scenes? Participants consistently matched specific auditory AM rates to diverse visual scenes (nature, urban, and indoor). A correlation analysis indicated that higher subjective density ratings were associated with faster AM-rate matches. Furthermore, both the density ratings and AM-rate matches were relatively scale invariant, suggesting that the underlying crossmodal association is between visual coding of object-based density and auditory coding of AM rate. Based on these results, we hypothesized that concurrently presented fast (7 Hz) or slow (2 Hz) AM-rates might influence how visual attention is allocated to dense or sparse regions within a scene. We tested this hypothesis by monitoring eye movements while participants examined scenes for a subsequent memory task. To determine whether fast or slow sounds guided eye movements to specific spatial frequencies, we computed the maximum contrast energy at each fixation across 12 spatial frequency bands ranging from 0.06–10.16 cycles/degree. We found that the fast sound significantly guided eye movements toward regions of high spatial frequency, whereas the slow sound guided eye movements away from regions of high spatial frequency. This suggests that faster sounds may promote a local scene scanning strategy, acting as a ‘filter’ to individuate objects within dense regions. Our results suggest that auditory AM rate and visual object density are crossmodally associated, and that this association can modulate visual inspection of scenes.