Perceptual exploration and intersensory merging in a visual–haptic Necker cube

Abstract

Perceiving the three-dimensional structure of an object often involves merging vision and haptics over extended periods of exploration. An interesting feature of this process is that as exploration progresses, new information may require changing how the two sensory signals are treated. We tested whether (1) bimodal interactions during extended observation can adjust to changes in the quality of unimodal sensory signals; and (2) there are specific temporal constraints on these adjustments. When viewing a three-dimensional Necker cube with one eye, participants experience illusory reversals even while they feel the cube with their hands. This surprising property makes the visual–haptic Necker cube an excellent model to investigate bimodal processes when previously consistent bimodal signals change in quality over time or begin to conflict. Our participants reported reversals while they viewed the cube and, at the same time, they either held it with two-finger grips, felt it with while their hands remained stationary, or actively explored it by moving one hand. Consistent with a multisensory approach to three-dimensional form perception, touch had a clear effect on both the number and the duration of illusory percepts. Additionally, when observers alternated between stationary and moving periods during exploration, transitions from stationary to moving-hand haptics played a crucial role in inhibiting illusory reversals. A temporal analysis of the probability of first reversals occurring after different types of motor transition revealed a “vetoing window” initiating approximately 2 s after the transition and lasting at least another 1–2 s. These results suggest that intersensory merging is sensitive to changes in the quality of unimodal signals during exploration, and that adjustments caused by these changes are characterized by specific temporal dynamics.