Multimodal Cues for Object Manipulation in Augmented and Virtual Environments

Abstract

The purpose of this work is to investigate the role of multimodal, especially auditory displays on human manipulation in augmented environments. We use information from all our sensory modalities when interacting in natural environments. Despite differences among the senses, we use them in concert to perceive and interact with multimodally specified objects and events. Traditionally, human-computer interaction has focused on graphical displays, thus not taking advantage of the richness of human senses and skills developed though interaction with the physical world [1]. Virtual environments have the potential to integrate all sensory modalities, to present the user with multiple inputs and outputs, and to allow the user to directly acquire and manipulate augmented or virtual objects. With the increasing availability of haptic and auditory displays, it is important to understand the complex relationships amongst different sensory feedback modalities and how they affect performance when interacting with augmented and virtual objects. Background and motivation for this research, questions and hypotheses, and some preliminary results are presented. A plan for future experiments is proposed.