Motion Parallax Cues Research Articles

Investigating visual determinants of visuomotor performance in virtual reality

We report the relative efficiency of visually guided movement in virtual reality (VR) compared to that in the real world using a standardised visuomotor task based on Fitts’ tapping. Haptic cues were veridical across both displays to ensure that any differences in performance could be attributed to characteristics of the visual display. The presence of binocular cues, and of monocular surface texture and motion parallax cues was manipulated across four ordinal visual conditions: binocular + texture + motion, monocular + texture + motion, monocular + motion, monocular. Binocular cues were found to have a similar benefit for VR and real-world performance. Surface texture cues were found to have no benefit for either display. Motion parallax cues were found to have a similar benefit for VR and real-world performance. Overall, task performance in VR was lower than in the real world due to both a slowing of movements and a decrease in their precision. Taken together, these results did not provide evidence for deficiencies in the presentation of binocular, texture or motion parallax cues in VR and suggests a non-visual locus for the deficit in visuomotor performance in VR. Limitations in the accuracy and precision of hand tracking are sufficient to explain the observed decrement in performance in VR.

A neural mechanism for detecting object motion during self-motion.

Detection of objects that move in a scene is a fundamental computation performed by the visual system. This computation is greatly complicated by observer motion, which causes most objects to move across the retinal image. How the visual system detects scene-relative object motion during self-motion is poorly understood. Human behavioral studies suggest that the visual system may identify local conflicts between motion parallax and binocular disparity cues to depth and may use these signals to detect moving objects. We describe a novel mechanism for performing this computation based on neurons in macaque middle temporal (MT) area with incongruent depth tuning for binocular disparity and motion parallax cues. Neurons with incongruent tuning respond selectively to scene-relative object motion, and their responses are predictive of perceptual decisions when animals are trained to detect a moving object during self-motion. This finding establishes a novel functional role for neurons with incongruent tuning for multiple depth cues.

Variations in visual sensitivity predict motion sickness in virtual reality

Severity of motion sickness varies across individuals. While some experience immediate symptoms, others seem relatively immune. We explored a potential explanation for such individual variability based on cue conflict theory. According to cue conflict theory, sensory signals that lead to mutually incompatible perceptual interpretations will produce physical discomfort. A direct consequence of such theory is that individuals with greater sensitivity to visual (or vestibular) sensory cues should show greater susceptibility, because they would be more likely to detect a conflict. Using virtual reality (VR), we assessed individual sensitivity to a number of visual cues and subsequently induced moderate levels of motion sickness using stereoscopic movies presented in the VR headset. We found that an observer’s sensitivity to motion parallax cues predicted severity of motion sickness symptoms. We also evaluated evidence for another reported source of variability in motion sickness severity in VR, namely sex, but found little support. We speculate that previously reported sex differences might have been due to poor personalization of VR displays, which default to male settings and introduce cue conflicts for the majority of females. Our results identify a sensitivity-based sensory predictor of motion sickness, which can be used to personalize VR experiences and mitigate discomfort.

The Effects of Depth Cues and Vestibular Translation Signals on the Rotation Tolerance of Heading Tuning in Macaque Area MSTd.

When the eyes rotate during translational self-motion, the focus of expansion (FOE) in optic flow no longer indicates heading, yet heading judgements are largely unbiased. Much emphasis has been placed on the role of extraretinal signals in compensating for the visual consequences of eye rotation. However, recent studies also support a purely visual mechanism of rotation compensation in heading-selective neurons. Computational theories support a visual compensatory strategy but require different visual depth cues. We examined the rotation tolerance of heading tuning in macaque area MSTd using two different virtual environments, a frontoparallel (2D) wall and a 3D cloud of random dots. Both environments contained rotational optic flow cues (i.e., dynamic perspective), but only the 3D cloud stimulus contained local motion parallax cues, which are required by some models. The 3D cloud environment did not enhance the rotation tolerance of heading tuning for individual MSTd neurons, nor the accuracy of heading estimates decoded from population activity, suggesting a key role for dynamic perspective cues. We also added vestibular translation signals to optic flow, to test whether rotation tolerance is enhanced by non-visual cues to heading. We found no benefit of vestibular signals overall, but a modest effect for some neurons with significant vestibular heading tuning. We also find that neurons with more rotation tolerant heading tuning typically are less selective to pure visual rotation cues. Together, our findings help to clarify the types of information that are used to construct heading representations that are tolerant to eye rotations.

Dynamic Corridor Illusion in Pigeons: Humanlike Pictorial Cue Precedence Over Motion Parallax Cue in Size Perception.

Depth information is necessary for perceiving the real size of objects at varying visual distances. To investigate to what extent this size constancy present in another vertebrate class, we addressed the two questions using pigeons: (a) whether pigeons see a corridor illusion based on size constancy and (b) whether pigeons prioritize pictorial cues over motion parallax cues for size constancy, like humans. We trained pigeons to classify target sizes on a corridor. In addition, we presented a dynamic version of corridor illusion in which the target and corridor moved side by side. Target speed was changed to manipulate motion parallax. With the static corridor, pigeons overestimated the target size when it was located higher, indicating that pigeons see a corridor illusion like humans. With the dynamic corridor, the pigeons overestimated the target size depending on target position, as in the static condition, but target speed did not affect their responses, indicating that the pictorial precedence also applies to pigeons. In a follow-up experiment using the same stimulus, we confirmed that humans perceive object size based on pictorial cues. These results suggest that size constancy characteristics are highly similar between pigeons and humans, despite the differences in their phylogeny and neural systems.

MannequinChallenge: Learning the Depths of Moving People by Watching Frozen People.

We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving (right). Existing methods for recovering depth for dynamic, non-rigid objects from monocular video impose strong assumptions on the objects' motion and may only recover sparse depth. In this paper, we take a data-driven approach and learn human depth priors from a new source of data: thousands of Internet videos of people imitating mannequins, i.e., freezing in diverse, natural poses, while a hand-held camera tours the scene (left). Because people are stationary, geometric constraints hold, thus training data can be generated using multi-view stereo reconstruction. At inference time, our method uses motion parallax cues from the static areas of the scenes to guide the depth prediction. We evaluate our method on real-world sequences of complex human actions captured by a moving hand-held camera, show improvement over state-of-the-art monocular depth prediction methods, and demonstrate various 3D effects produced using our predicted depth.

Dual focal plane augmented reality interactive display with gaze-tracker

Stereoscopic augmented reality (AR) displays have a fixed focus plane and they suffer from visual discomfort due to vergence-accommodation conflict (VAC). In this study, we demonstrated a biocular (i.e. common optics for two eyes and same images are shown to both eyes) two focal-plane based AR system with real-time gaze tracker, which provides a novel interactive experience. To mitigate VAC, we propose a see-through near-eye display mechanism that generates two separate virtual image planes at arm’s length depth levels (i.e. 25 cm and 50 cm). Our optical system generates virtual images by relaying two liquid crystal displays (LCDs) through a beam splitter and a Fresnel lens. While the system is limited to two depths and discontinuity occurs in the virtual scene, it provides correct focus cues and natural blur effect at the corresponding depths. This allows the user to distinguish virtual information through the accommodative response of the eye, even when the virtual objects overlap and partially occlude in the axial direction. The system also provides correct motion parallax cues within the movement range of the user without any need for sophisticated head trackers. A road scene simulation is realized as a convenient use-case of the proposed display so that a large monitor is used to create a background scene and the rendered content in the LCDs is augmented into the background. Field-of-view (FOV) is 60 × 36 degrees and the eye-box is larger than 100 mm, which is comfortable enough for two-eye viewing. The system includes a single camera-based pupil and gaze tracker, which is able to select the correct depth plane based on the shift in the interpupillary distance with user’s convergence angle. The rendered content can be distributed to both depth planes and the background scene simultaneously. Thus, the user can select and interact with the content at the correct depth in a natural and comfortable way. The prototype system can be used in tasks that demand wide FOV and multiple focal planes and as an AR and vision research tool.

Motion Parallax Improves Object Recognition in the Presence of Clutter in Simulated Prosthetic Vision.

PurposeEfficacy of current visual prostheses in object recognition is limited. Among various limitations to be addressed, such as low resolution and low dynamic range, here we focus on reducing the impact of background clutter on object recognition. We have proposed the use of motion parallax via head-mounted camera lateral scanning and computationally stabilizing the object of interest (OI) to support neural background decluttering. Simulations in head-mounted displays (HMD), mimicking the proposed effect, were used to test object recognition in normally sighted subjects.MethodsImages (24° field of view) were captured from multiple viewpoints and presented at a low resolution (20 × 20). All viewpoints were centered on the OI. Experimental conditions (2 × 3) included clutter (with or without) × head scanning (single viewpoint, 9 coherent viewpoints corresponding to subjects' head positions, and 9 randomly associated viewpoints). Subjects used lateral head movements to view OIs in the HMD. Each object was displayed only once for each subject.ResultsThe median recognition rate without clutter was 40% for all head scanning conditions. Performance with synthetic background clutter dropped to 10% in the static condition, but it was improved to 20% with the coherent and random head scanning (corrected P = 0.005 and P = 0.049, respectively).ConclusionsBackground decluttering using motion parallax cues but not the coherent multiple views of the OI improved object recognition in low-resolution images. The improvement did not fully eliminate the impact of background.Translational RelevanceMotion parallax is an effective but incomplete decluttering solution for object recognition with visual prostheses.

User Behavior and the Importance of Stereo for Depth Perception in Fish Tank Virtual Reality

Abstract Since one of the most important aspects of a Fish Tank Virtual Reality (FTVR) system is how well it provides the illusion of depth to users, we present a study that evaluates users' depth perception in FTVR systems using three tasks. The tasks are based on psychological research on human vision and depth judgments common in VR applications. We find that participants do not perform well under motion parallax cues only, when compared with stereo only or a combination of both kinds of cues. Measurements of participants' head movement during each task prove valuable in explaining the experimental findings. We conclude that FTVR users rely on stereopsis for depth perception in FTVR environments more than they do on motion parallax, especially for tasks requiring depth acuity.

Use of cues in virtual reality depends on visual feedback

3D motion perception is of central importance to daily life. However, when tested in laboratory settings, sensitivity to 3D motion signals is found to be poor, leading to the view that heuristics and prior assumptions are critical for 3D motion perception. Here we explore an alternative: sensitivity to 3D motion signals is context-dependent and must be learned based on explicit visual feedback in novel environments. The need for action-contingent visual feedback is well-established in the developmental literature. For example, young kittens that are passively moved through an environment, but unable to move through it themselves, fail to develop accurate depth perception. We find that these principles also obtain in adult human perception. Observers that do not experience visual consequences of their actions fail to develop accurate 3D motion perception in a virtual reality environment, even after prolonged exposure. By contrast, observers that experience the consequences of their actions improve performance based on available sensory cues to 3D motion. Specifically, we find that observers learn to exploit the small motion parallax cues provided by head jitter. Our findings advance understanding of human 3D motion processing and form a foundation for future study of perception in virtual and natural 3D environments.

Distinct spatial coordinate of visual and vestibular heading signals in macaque FEFsem and MSTd.

Precise heading estimate requires integration of visual optic flow and vestibular inertial motion originating from distinct spatial coordinates (eye- and head-centered, respectively). To explore whether the two heading signals may share a common reference frame along the hierarchy of cortical stages, we explored two multisensory areas in macaques: the smooth pursuit area of the frontal eye field (FEFsem) closer to the motor side, and the dorsal portion of medial superior temporal area (MSTd) closer to the sensory side. In both areas, vestibular signals are head-centered, whereas visual signals are mainly eye-centered. However, visual signals in FEFsem are more shifted towards the head coordinate compared to MSTd. These results are robust being largely independent on: (1) smooth pursuit eye movement, (2) motion parallax cue, and (3) behavioral context for active heading estimation, indicating that the visual and vestibular heading signals may be represented in distinct spatial coordinate in sensory cortices.

Motion parallax in stereo 3D

Binocular disparity is the main depth cue that makes stereoscopic images appear 3D. However, in many scenarios, the range of depth that can be reproduced by this cue is greatly limited and typically fixed due to constraints imposed by displays. For example, due to the low angular resolution of current automultiscopic screens, they can only reproduce a shallow depth range. In this work, we study the motion parallax cue, which is a relatively strong depth cue, and can be freely reproduced even on a 2D screen without any limits. We exploit the fact that in many practical scenarios, motion parallax provides sufficiently strong depth information that the presence of binocular depth cues can be reduced through aggressive disparity compression. To assess the strength of the effect we conduct psycho-visual experiments that measure the influence of motion parallax on depth perception and relate it to the depth resulting from binocular disparity. Based on the measurements, we propose a joint disparity-parallax computational model that predicts apparent depth resulting from both cues. We demonstrate how this model can be applied in the context of stereo and multiscopic image processing, and propose new disparity manipulation techniques, which first quantify depth obtained from motion parallax, and then adjust binocular disparity information accordingly. This allows us to manipulate the disparity signal according to the strength of motion parallax to improve the overall depth reproduction. This technique is validated in additional experiments.

Depth discrimination from occlusions in 3D clutter

Objects such as trees, shrubs, and tall grass consist of thousands of small surfaces that are distributed over a three-dimensional (3D) volume. To perceive the depth of surfaces within 3D clutter, a visual system can use binocular stereo and motion parallax. However, such parallax cues are less reliable in 3D clutter because surfaces tend to be partly occluded. Occlusions provide depth information, but it is unknown whether visual systems use occlusion cues to aid depth perception in 3D clutter, as previous studies have addressed occlusions for simple scene geometries only. Here, we present a set of depth discrimination experiments that examine depth from occlusion cues in 3D clutter, and how these cues interact with stereo and motion parallax. We identify two probabilistic occlusion cues. The first is based on the fraction of an object that is visible. The second is based on the depth range of the occluders. We show that human observers use both of these occlusion cues. We also define ideal observers that are based on these occlusion cues. Human observer performance is close to ideal using the visibility cue but far from ideal using the range cue. A key reason for the latter is that the range cue depends on depth estimation of the clutter itself which is unreliable. Our results provide new fundamental constraints on the depth information that is available from occlusions in 3D clutter, and how the occlusion cues are combined with binocular stereo and motion parallax cues.

How various aspects of motion parallax influence distance judgments, even when we think we are standing still.

It is well known that when we intentionally make large head movements, the resulting motion parallax helps us judge objects' distances. The information about distance could be obtained in various ways: from the changes in the object's position with respect to ourselves, from the changes in its orientation relative to the line of sight, and from the relative retinal motion between the target's image and that of the background. We explore here whether these motion parallax cues are used when we think we are standing still. To answer this question we asked subjects to indicate the position of a virtual target with their unseen finger. The position and the size of the target changed across trials. There were pairs of trials in which the same target was presented at the same location, except that one or more of the three motion parallax cues indicated that the target was either 10 cm closer or 10 cm farther away than the 'true' distance. Any systematic difference between the positions indicated for the closer and further targets of such pairs indicates that the cues in question influence subjects' judgments. The results show that motion parallax cues have a detectable influence on our judgments, even when the head only moves a few millimeters. Relative retinal image motion has the clearest effect. Subjects did not move their head differently when we presented the targets to only one eye in order to increase the benefit of considering motion parallax.

Role of visual and non-visual cues in constructing a rotation-invariant representation of heading in parietal cortex.

As we navigate through the world, eye and head movements add rotational velocity patterns to the retinal image. When such rotations accompany observer translation, the rotational velocity patterns must be discounted to accurately perceive heading. The conventional view holds that this computation requires efference copies of self-generated eye/head movements. Here we demonstrate that the brain implements an alternative solution in which retinal velocity patterns are themselves used to dissociate translations from rotations. These results reveal a novel role for visual cues in achieving a rotation-invariant representation of heading in the macaque ventral intraparietal area. Specifically, we show that the visual system utilizes both local motion parallax cues and global perspective distortions to estimate heading in the presence of rotations. These findings further suggest that the brain is capable of performing complex computations to infer eye movements and discount their sensory consequences based solely on visual cues.

Unilateral range finding in diving beetle larvae

One of the biggest challenges that predators, such as the larvae of the diving beetle Thermonectus marmoratus (Coleoptera: Dytiscidae), are faced with is to accurately assess the distance of their prey. Most animals derive distance information from disparities of images that are viewed from different angles, from information that is obtained from well-controlled translational movements (motion parallax) or from the image size of known objects. Using a behavioral assay we demonstrated that T. marmoratus larvae continue to accurately strike at artificial prey, even if none of these typical distance estimation cues are available to them. Specifically, we excluded bilateral binocular stereopsis by occlusion, confounded possible motion parallax cues with an artificially moving prey, and excluded the possibility that beetle larvae simply approached their targets based on known prey size by presenting different prey sizes. Despite these constraints, larvae consistently struck our artificial targets from a distance of ~4.5 mm. Based on these findings we conclude that T. marmoratus likely employ an unusual mechanism to accurately determine prey distances, possibly mediated by the object-distance-dependent activation of specific subsets of their many-tiered and peculiarly positioned photoreceptors.

Psychophysical point: A disc tends to become a point when Weber’s law fails

In three experiments, we showed that the visual system treats a dot somewhat like a geometrical point, which has a location but no area. We represented a "point" or "dot" with a small disc (diameter of 0.08º of visual angle), and a "disc" with a larger disc (diameter of 1.5º). The Weber fraction of the dot was larger than that of the disc. In Experiment 1, the relative retinal image size cues for depth for the dot and the disc were placed in conflict with the motion parallax cue. We found that the dot indicated the positions defined by the motion parallax cue better than the disc did. In Experiment 2, we placed a constant retinal image size in conflict with convergence eye movements. We found that a binocularly fused dot appeared to move in depth with convergence eye movement, whereas a fused disc appeared to move less. In Experiment 3, we examined the apparent sizes of the afterimages of a dot and a disc and found that Emmert's law failed for the dot afterimage; the apparent size of the dot afterimage changed very little for different distances-as though it had no area-whereas the apparent size of the disc afterimage changed by an extent predicted by Emmert's law. The differences in the dot and disc conditions could not be explained by the differences in the Weber fractions alone.

Joint Representation of Depth from Motion Parallax and Binocular Disparity Cues in Macaque Area MT

Perception of depth is based on a variety of cues, with binocular disparity and motion parallax generally providing more precise depth information than pictorial cues. Much is known about how neurons in visual cortex represent depth from binocular disparity or motion parallax, but little is known about the joint neural representation of these depth cues. We recently described neurons in the middle temporal (MT) area that signal depth sign (near vs far) from motion parallax; here, we examine whether and how these neurons also signal depth from binocular disparity. We find that most MT neurons in rhesus monkeys (Macaca Mulatta) are selective for depth sign based on both disparity and motion parallax cues. However, the depth-sign preferences (near or far) are not always aligned: 56% of MT neurons have matched depth-sign preferences ("congruent" cells) whereas the remaining 44% of neurons prefer near depth from motion parallax and far depth from disparity, or vice versa ("opposite" cells). For congruent cells, depth-sign selectivity increases when disparity cues are added to motion parallax, but this enhancement does not occur for opposite cells. This suggests that congruent cells might contribute to perceptual integration of depth cues. We also found that neurons are clustered in MT according to their depth tuning based on motion parallax, similar to the known clustering of MT neurons for binocular disparity. Together, these findings suggest that area MT is involved in constructing a representation of 3D scene structure that takes advantage of multiple depth cues available to mobile observers.

Perceived depth magnitude with a combination of motion parallax and binocular disparity cues

Perceived depth magnitude with a combination of motion parallax and binocular disparity cues

Full parallax three-dimensional computer generated hologram with occlusion effect using ray casting technique

Holographic display is capable of reconstructing the whole optical wave field of a three-dimensional (3D) scene. It is the only one among all the 3D display techniques that can produce all the depth cues. With the development of computing technology and spatial light modulators, computer generated holograms (CGHs) can now be used to produce dynamic 3D images of synthetic objects. Computation holography becomes highly complicated and demanding when it is employed to produce real 3D images. Here we present a novel algorithm for generating a full parallax 3D CGH with occlusion effect, which is an important property of 3D perception, but has often been neglected in fully computed hologram synthesis. The ray casting technique, which is widely used in computer graphics, is introduced to handle the occlusion issue of CGH computation. Horizontally and vertically distributed rays are projected from each hologram sample to the 3D objects to obtain the complex amplitude distribution. The occlusion issue is handled by performing ray casting calculations to all the hologram samples. The proposed algorithm has no restriction on or approximation to the 3D objects, and hence it can produce reconstructed images with correct shading effect and no visible artifacts. Programmable graphics processing unit (GPU) is used to perform parallel calculation. This is made possible because each hologram sample belongs to an independent operation. To demonstrate the performance of our proposed algorithm, an optical experiment is performed to reconstruct the 3D scene by using a phase-only spatial light modulator. We can easily perceive the accommodation cue by focusing our eyes on different depths of the scene and the motion parallax cue with occlusion effect by moving our eyes around. The experiment result confirms that the CGHs produced by our algorithm can successfully reconstruct 3D images with all the depth cues.