Listeners as spectators? Audio-visual integration improves music performer identification

Abstract

Listeners take for granted their capacity to distinguish between musical instruments, and their ability to discriminate between performers playing the same instrument by their sound alone. Sound perception is usually considered a purely auditory process, but there is significant debate on how auditory and visual information are integrated during listening. Two experiments examined how listeners perceive individual performers. Saxophonists ( n = 5) performed three jazz standards for an audio and video recording. Experiment 1 investigated listeners’ ability to identify individual saxophonists by ear. Listeners heard one saxophonist, and were later asked to identify him from a line-up of saxophonists. Only 52% of listeners could correctly identify their target. Experiment 2 explored the integration of cross-modal sensory experiences (audio and visual) in saxophonist identification. Participants either watched a silent video clip of a saxophonist playing and matched it to an audio clip of the same performer, or heard an audio clip of a saxophonist and matched it to a silent video clip. Listener/viewers reliably identified their target saxophonists, and results suggest that listeners combine cross-modal perceptions to identify individual performers, and can use the information about a performer in one modality and match it to the same performer in another modality.