Rhythm perception is shared between audio and haptics

Corentin Bernard,Michaël Wiertlewski,Jocelyn Monnoyer,Sølvi Ystad

doi:10.1038/s41598-022-08152-w

Abstract

A surface texture is perceived through both the sound and vibrations produced while being explored by our fingers. Because of their common origin, both modalities have a strong influence on each other, particularly at above 60 Hz for which vibrotactile perception and pitch perception share common neural processes. However, whether the sensation of rhythm is shared between audio and haptic perception is still an open question. In this study, we show striking similarities between the audio and haptic perception of rhythmic changes, and demonstrate the interaction of both modalities below 60 Hz. Using a new surface-haptic device to synthesize arbitrary audio-haptic textures, psychophysical experiments demonstrate that the perception threshold curves of audio and haptic rhythmic gradients are the same. Moreover, multimodal integration occurs when audio and haptic rhythmic gradients are congruent. We propose a multimodal model of rhythm perception to explain these observations. These findings suggest that audio and haptic signals are likely to be processed by common neural mechanisms also for the perception of rhythm. They provide a framework for audio-haptic stimulus generation that is beneficial for nonverbal communication or modern human-machine interfaces.

Highlights

A surface texture is perceived through both the sound and vibrations produced while being explored by our fingers
Touching these haptic stimuli produces the sensation of bumps that becomes closer or more distant from each other, like accelerating or decelerating rhythmic patterns. The perception of these haptic gradients is here investigated by a psychophysical experiment, whose results are compared with the literature on auditory perception
Audio and haptic perception are known to interact for pitches above 100 Hz, and in this series of experiments we demonstrated that this interaction extends to the perception of rhythm and its temporal evolution

Summary

Introduction

A surface texture is perceived through both the sound and vibrations produced while being explored by our fingers. While psychophysical experiments demonstrate high-level interactions between audio and tactile sensory s ystems[6–9], neuroimaging studies suggest that these interactions occur in early sensory areas[10–12] These experiments reveal strong interactions and common neural processes for vibrotactile perception and pitch perception, for frequencies above 60 Hz. audio-tactile interactions with lower frequency content, associated with rhythm, in particular rhythmic changes, are rarely investigated. Touching these haptic stimuli produces the sensation of bumps that becomes closer or more distant from each other, like accelerating or decelerating rhythmic patterns The perception of these haptic gradients is here investigated by a psychophysical experiment, whose results are compared with the literature on auditory perception.

Methods

Results

Conclusion