Letter of the guest editors of the special issue on “Music as a multimodal experience”.

Abstract

It is our pleasure to introduce a collection of papers brought together under the heading of as a Multimodal Experience. This issue aims to bring to the foreground recent developments in our understanding of music as a multimodal experience. Music listening generally concerns multiple modalities in the sense that other modalities than hearing are simultaneously present and influence how we perceive and engage with music. This multimodality includes visual aspects of music production or multimedia presentations of music, tactile experiences of frequency, intensity, and rhythm, kinematic responses to music, and so forth. However, even if we isolate the presentation mode of music to auditory, multiple sense-modalities are nevertheless involved in the perception of music, activated through cross-modal correspondences or cross-modal associations.To investigate the role of these cross-modal correspondences in the perception and cognition of music, a British Academy-funded network was established between researchers at The University of Sheffield, United Kingdom, Tel Aviv University, Israel, and The Hebrew University of Jerusalem, also in Israel. A number of research projects were initiated within this network, one of which is reported in this special issue (contribution by Timmers & Li). Additionally, the network organized the International Conference on the Multimodal Experience of Music, which was held March 23-25, 2015, at The University of Sheffield, United Kingdom. The papers brought together in this special issue provide a small, but relevant, cross-division of the wide ranging studies and topics presented at the conference (for further reading, see the ICMEM Proceedings)1.Firstly, Walker contextualizes research on cross-modal associations with musical parameters within a broader theoretical framework of cross-sensory correspondences. This is followed by four studies that investigate cross-modal associations that may occur while listening to music. The contribution by Miller explores the presence of pitch-related kinesthetic vocal sensations during music listening (i.e., in the vocal tract, ears, eustachian tubes), examining the hypothesis that auditory and vocal processes share peripheral substrates. The study by Metcalfe investigates the relationship between musical attributes (tempo, intensity, and jitter in periodicity) and spontaneous adaptations in walking when listening to the music. It suggests an integrative model in which both arousal and rhythmic entrainment interact in modulating walking speed. Timmers and Li investigate factors contributing to a cross-modally induced perceptual illusion: The perception of the horizontal location of brief sounds is influenced by the pitch of the sound. An illusion of high sounds coming from the right and low sounds coming from the leftis stronger for participants with higher levels of musical training. It is also stronger for pianist than for flutists, however, only after performing their respective instrument. Isben and Krumhansl explore the possible mediating role of the perception of emotion in music for music-color associations. They present clear analogies between attributed emotions to colors, attributed emotions to music, and attributed colors to music, which is further supported by predicting music-color associations based on color- emotion mappings.The second set of studies presented in the issue examine the contribution of different modalities to the perception of music presented in a unimodal versus a multimodal fashion. Slater and Marozeau investigate whether the addition of tactile cues can facilitate the segregation of a melody within a context in which two melodies are interleaved and (without assistance for selective attention) perceptually integrated. …