Bouncing human vs. bouncing balls: Synchrony perception between auditory rhythm and different naturalistic visual motions

Abstract

Temporal integration of auditory and visual information is often examined in scenarios where sounds are produced by the observed action (e.g., impact sounds paired with drumming movements). The present study, however, investigated synchrony perception between concurrent auditory and visual streams when the sounds are not consequent upon the movement (e.g., watching a dancer moving to the music), and how visual motion velocity affects this perception. In Experiment 1, participants judged synchrony at the bounce between a bouncing point-light figure (PLF) and a regular auditory rhythm in four visual conditions: human PLF bouncing with human motion, human PLF bouncing with ball motion (without damping), non-human PLF bouncing with human motion, and non-human PLF bouncing with ball motion. Regardless of the figure, a greater delay of the auditory stream was found for synchrony perception when viewing movements of ball motion than of human motion. This suggests a shift of point of subjective synchrony (PSS) towards the maximal velocity change within the observed trajectory, which occurs later in ball motion than in human motion. Experiment 2 measured synchrony perception either at the bounce or at the highest position of the same movement. The effect of motion on PSS was found again for synchrony judgement at the bounce, but not at the highest position, where velocity change was similar in both motions. Together these results demonstrate the effect of visual velocity cue — specifically the point of peak velocity change in the trajectory — on audiovisual synchrony perception in a continuous naturalistic movement.