Effects of visual and auditory cues on haptic illusions for active and passive touches in mixed reality

Abstract

The haptic illusion literature suggests that individuals’ haptic experiences rely not only on tactile signals, but on visual and auditory stimulation as well. Given that mixed reality (MR) technologies often harness realistic visual and auditory but not haptic stimulation, it is important to understand how visual and auditory signals shape haptic experiences. We examined whether visual and auditory cues in MR can enhance tactile sensation, including perceptions of roughness and stiffness on virtual gadgets, in the absence of haptic signals. A laboratory experiment showed that tactile perception of mobile gadgets in motion was influenced by both visual and auditory cues when the motion of the gadget was generated by participants (i.e., active touch), but only by auditory cues when participants observed the gadget moving (i.e., passive touch). The results indicate that visual and auditory cues can be employed to facilitate haptic experiences of virtual objects in virtual reality, with auditory cues being potentially applicable to a broader context than visual cues.