Effects of Auditory Feedback on Visually-Guided Movement in Real and Virtual Space

Abstract

There is increasing interest in the use of virtual reality (VR) for training and simulation. While visual and auditory VR displays have improved markedly, the presentation of haptic (tactile and proprioceptive) cues remains challenging and poses limits on users’ manual interaction with the virtual environment. We examined interaction with a physical touchscreen and with a virtual touchscreen in VR using a standardized Fitts’ tapping task. We examined the effect on the performance of cues to touch of the screen (contact feedback) or to the accuracy of pointing to targets (error feedback) delivered via the auditory modality. For the touchscreen, error feedback reduced target misses while maintaining the efficiency of movement (throughput). In VR, error feedback reduced misses but also reduced throughput due to a disproportionate slowing of movements. Auditory cues to contact significantly improved throughput in VR but not for the touchscreen, possibly because the touchscreen provided robust tactile and proprioceptive cues to contact. The improvement in throughput with auditory contact feedback in VR was of a similar magnitude to that previously reported for haptic contact feedback, but smaller than that reported for veridical haptic cues. These results suggest that auditory stimuli are able to substitute in a limited manner in the absence of tactile or proprioceptive stimulation in VR environments.