Sonification of virtual and real surface tapping: evaluation of behavior changes, surface perception and emotional indices

Abstract

© 2015 IEEE. The audio-feedback resulting from object interaction provides information about the material of the surface and about one's own motor behavior. With the current developments in interactive sonification, it is now possible to digitally change this audio-feedback, and thus, the use of interactive sonification becomes a compelling approach to shape tactile surface interactions. Here, we present a prototype for a sonic interactive surface, capable of delivering surface tapping sounds in real-time, when triggered by the users' taps on a real surface or on an imagined, virtual surface. In this system, the delivered audio-feedback can be varied so that the heard tapping sounds correspond to different applied strength during tapping. Here, we also propose a multi-dimensional measurement approach to evaluate user experiences of multi-modal interactive systems. We evaluated our system by looking at the effect of the altered tapping sounds on emotional action-related responses, users' way of interacting with the surface, and perceived surface hardness. Results show the influence of the sonification of tapping at all levels: emotional, behavioral and perceptual. These results have implications in the design of interactive sonification displays and tangible auditory interfaces aiming to change perceived and subsequent motor behaviour, as well as perceived material properties.