Interaction of sound and sight during action perception: Evidence for shared modality-dependent action representations

Abstract

Seeing or hearing manual actions activates the mirror neuron system, i.e., specialized neurons within motor areas which fire not only when an action is performed but also when it is passively perceived. Although it has been shown that mirror neurons respond to either action-specific vision or sound, it remains a topic of debate whether and how vision and sound interact during action perception. Here we used transcranial magnetic stimulation to explore multimodal interactions in the human motor system, namely at the level of the primary motor cortex (M1). Corticomotor excitability in M1 was measured while subjects perceived unimodal visual (V), unimodal auditory (A), or multimodal (V+A) stimuli of a simple hand action. In addition, incongruent multimodal stimuli were included, in which incongruent vision or sound was presented simultaneously with the auditory or visual action stimulus. A selective response increase was observed to the congruent multimodal stimulus as compared to the unimodal and incongruent multimodal stimuli. These findings speak in favour of 'shared' action representations in the human motor system that are evoked in a 'modality-dependent' way, i.e., they are elicited most robustly by the simultaneous presentation of congruent auditory and visual stimuli. Multimodality in the perception of hand movements bears functional similarities to speech perception, suggesting that multimodal convergence is a generic feature of the mirror system which applies to action perception in general.