Multisensory ambiguities in impossible worlds: Assessing auditory, visual, and motor contributions to the representation of space

Abstract

We conducted behavioral experiments on visual, auditory, and motor contributions to the human representation of space in virtual reality environments using an ‘impossible-worlds paradigm’. The experiments were run with an omnidirectional locomotion input device, the ‘Virtusphere’, which is a rotatable 10-foot hollow sphere that allows a subject inside to walk in any direction for any distance, while immersed in a virtual environment. Both the rotation of the sphere and the movement of a subject’s head were tracked to process the subject’s view within the virtual environment presented on a head-mounted display. Auditory features were dynamically processed in order to exactly align sound sources and visual objects. Using this experimental setup the subjects were presented with ‘impossible worlds’, i.e., virtual environments with geometrical and topological properties, which are physically not possible. In previous experiments we have shown that subjects are able to navigate inside these impossible worlds (Zetzsche et al., 2009), despite the fact that different interpretations of their spatial structure are in conflict, since there is no single (physically plausible) interpretation accounting for all sensory perceptions of the subjects. In the present study we manipulated these physically ‘impossible’ properties either in the visual or in the auditory domain (so that each modality supports one of the possible interpretations) and assessed how these manipulations affected the subjects’ internal representations of space. We discuss our results with respect to auditory, visual, and motor contributions to the internal spatial representation, the interaction of modalities, and the implication on the notion of motor action as a linking element between the senses.