A vibrotactile music system based on sensory substitution.

Abstract

The idea of the project reported here was to design a system that builds on touch to enable people with severe hearing impairments to “listen” to music through a process called sensory substitution. The goal was to transform the auditory parameter space into one that is adequate for haptic perception. The approach reported here builds on (i) the design of a haptic display, a tabletop device with 8–24 actuators that can be driven individually, (ii) machine learning algorithms, and (iii) a psychophysical study to determine which music cues can be perceived through touch. The latter was necessary because vibrotactile perception is not yet well understood in the context of music perception. The double-blind study analyzes how vibrotactile stimuli contribute to the perception, cognition, and distinction of sounds in human participants who have been trained versus those who have not. In order to ensure that normal-hearing participants could not hear sounds radiated from the haptic display, sound isolating headphones were used to playback pink noise during the experiment.