Crossmodal correspondence between haptic shape and sound

Abstract

Accumulating evidence has demonstrated the existence of non-arbitrary correspondence between sensory modalities. For example, studies report non-arbitrary relationship between visual shape and sound: people tend to label a round shape 'bouba' and a spiky shape 'kiki' (Köhler, 1947; Ramachandran & Hubbard, 2001). Since such relationship is well established mostly in the visual and auditory domain, the present study aimed to examine the correspondence between the haptic and auditory modality, which has not been illuminated much in previous studies. A three-dimensional, round and a spiky haptic stimulus were generated with a parametric shape model (Lee Masson et al., 2016), and linear interpolation was carried out on the coordinates of the two stimuli to generate five more stimuli in between. As a result, there were total seven haptic stimuli, of which the roundness/spikiness dimension was equi-distantly manipulated. Multidimensional scaling analysis confirmed that physical spaces conformed to perceptual space, showing the effectiveness of the manipulation of the physical parameters. For the auditory stimuli, we used Haskins Articulatory Synthesizer to generate two vowel sounds by manipulating the height and frontness of the tongue body position (e.g. /a/, /i/). These sounds were shown to be associated with either round or spiky visual shape (Kwak et al., iMRF 2017). On each trial of the experiment, participants (N=19) had to palpate a haptic stimulus for 5 seconds, after which it was taken away by the experimenter. Then the two sounds were presented sequentially, and the participants had to choose the sound that better matched the object. Results showed the main effect of shape: the roundest object was associated with /a/, whereas the spikiest object was associated with /i/, and responses for the objects in between were modulated according to the roundness/spikiness dimension. These results suggest that the correspondence between shape and sound extends to haptics and audition. Meeting abstract presented at VSS 2018