3D Motion Perception Research Articles

A Framework for the Continuous Evaluation of 3D Motion Perception in Virtual Reality.

We present a novel framework for the detection and continuous evaluation of 3D motion perception by deploying a virtual reality environment with built-in eye tracking. We created a biologically-motivated virtual scene that involved a ball moving in a restricted Gaussian random walk against a background of 1/f noise. Sixteen visually healthy participants were asked to follow the moving ball while their eye movements were monitored binocularly using the eye tracker. We calculated the convergence positions of their gaze in 3D using their fronto-parallel coordinates and linear least-squares optimization. Subsequently, to quantify 3D pursuit performance, we employed a first-order linear kernel analysis known as the Eye Movement Correlogram technique to separately analyze the horizontal, vertical and depth components of the eye movements. Finally, we checked the robustness of our method by adding systematic and variable noise to the gaze directions and re-evaluating 3D pursuit performance. We found that the pursuit performance in the motion-through depth component was reduced significantly compared to that for fronto-parallel motion components. We found that our technique was robust in evaluating 3D motion perception, even when systematic and variable noise was added to the gaze directions. The proposed framework enables the assessment of 3D Motion perception by evaluating continuous pursuit performance through eye-tracking. Our framework paves the way for a rapid, standardized and intuitive assessment of 3D motion perception in patients with various eye disorders.

Task feedback suggests a post-perceptual component to serial dependence.

Decisions across a range of perceptual tasks are biased toward past stimuli. Such serial dependence is thought to be an adaptive low-level mechanism that promotes perceptual stability across time. However, recent studies suggest post-perceptual mechanisms may also contribute to serially biased responses, calling into question a single locus of serial dependence and the nature of integration of past and present sensory inputs. We measured serial dependence in the context of a three-dimensional (3D) motion perception task where uncertainty in the sensory information varied substantially from trial to trial. We found that serial dependence varied with stimulus properties that impact sensory uncertainty on the current trial. Reduced stimulus contrast was associated with an increased bias toward the stimulus direction of the previous trial. Critically, performance feedback, which reduced sensory uncertainty, abolished serial dependence. These results provide clear evidence for a post-perceptual locus of serial dependence in 3D motion perception and support the role of serial dependence as a response strategy in the face of substantial sensory uncertainty.

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Scene flow tracks the 3D motion of each point in adjacent point clouds. It provides fundamental 3D motion perception for autonomous driving and server robot. Although red green blue depth (RGBD) camera or light detection and ranging (LiDAR) capture discrete 3D points in space, the objects and motions usually are continuous in the macroworld. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the generative adversarial networks (GAN) is utilized to self‐learn 3D scene flow without ground truth. The fake point cloud is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset show that our method realizes promising results. Just as human, the proposed method can identify the similar local structures of two adjacent frames even without knowing the ground truth scene flow. Then, the local correspondence can be correctly estimated, and further the scene flow is correctly estimated. An interactive preprint version of the article can be found here:https://www.authorea.com/doi/full/10.22541/au.163335790.03073492.

Spatiotemporal integration of isolated binocular three-dimensional motion cues

Two primary binocular cues—based on velocities seen by the two eyes or on temporal changes in binocular disparity—support the perception of three-dimensional (3D) motion. Although these cues support 3D motion perception in different perceptual tasks or regimes, stimulus cross-cue contamination and/or substantial differences in spatiotemporal structure have complicated interpretations. We introduce novel psychophysical stimuli which cleanly isolate the cues, based on a design introduced in oculomotor work (Sheliga, Quaia, FitzGibbon, & Cumming, 2016). We then use these stimuli to characterize and compare the temporal and spatial integration properties of velocity- and disparity-based mechanisms. On average, temporal integration of velocity-based cues progressed more than twice as quickly as disparity-based cues; performance in each pure-cue condition saturated at approximately 200 ms and approximately 500 ms, respectively. This temporal distinction suggests that disparity-based 3D direction judgments may include a post-sensory stage involving additional integration time in some observers, whereas velocity-based judgments are rapid and seem to be more purely sensory in nature. Thus, these two binocular mechanisms appear to support 3D motion perception with distinct temporal properties, reflecting differential mixtures of sensory and decision contributions. Spatial integration profiles for the two mechanisms were similar, and on the scale of receptive fields in area MT. Consistent with prior work, there were substantial individual differences, which we interpret as both sensory and cognitive variations across subjects, further clarifying the case for distinct sets of both cue-specific sensory and cognitive mechanisms. The pure-cue stimuli presented here lay the groundwork for further investigations of velocity- and disparity-based contributions to 3D motion perception.

Newtonian Predictions Are Integrated With Sensory Information in 3D Motion Perception

Because the motions of everyday objects obey Newtonian mechanics, perhaps these laws or approximations thereof are internalized by the brain to facilitate motion perception. Shepard’s seminal investigations of this hypothesis demonstrated that the visual system fills in missing information in a manner consistent with kinematic constraints. Here, we show that perception relies on internalized regularities not only when filling in missing information but also when available motion information is inconsistent with the expected outcome of a physical event. When healthy adult participants (Ns = 11, 11, 12, respectively, in Experiments 1, 2, and 3) viewed 3D billiard-ball collisions demonstrating varying degrees of consistency with Newtonian mechanics, their perceptual judgments of postcollision trajectories were biased toward the Newtonian outcome. These results were consistent with a maximum-likelihood model of sensory integration in which perceived target motion following a collision is a reliability-weighted average of a sensory estimate and an internal prediction consistent with Newtonian mechanics.

On the Aperture Problem of Binocular 3D Motion Perception.

Like many predators, humans have forward-facing eyes that are set a short distance apart so that an extensive region of the visual field is seen from two different points of view. The human visual system can establish a three-dimensional (3D) percept from the projection of images into the left and right eye. How the visual system integrates local motion and binocular depth in order to accomplish 3D motion perception is still under investigation. Here, we propose a geometric-statistical model that combines noisy velocity constraints with a spherical motion prior to solve the aperture problem in 3D. In two psychophysical experiments, it is shown that instantiations of this model can explain how human observers disambiguate 3D line motion direction behind a circular aperture. We discuss the implications of our results for the processing of motion and dynamic depth in the visual system.Dataset: osf.io/2j6sq.

Repetition Estimation

Visual repetition is ubiquitous in our world. It appears in human activity (sports, cooking), animal behavior (a bee’s waggle dance), natural phenomena (leaves in the wind) and in urban environments (flashing lights). Estimating visual repetition from realistic video is challenging as periodic motion is rarely perfectly static and stationary. To better deal with realistic video, we elevate the static and stationary assumptions often made by existing work. Our spatiotemporal filtering approach, established on the theory of periodic motion, effectively handles a wide variety of appearances and requires no learning. Starting from motion in 3D we derive three periodic motion types by decomposition of the motion field into its fundamental components. In addition, three temporal motion continuities emerge from the field’s temporal dynamics. For the 2D perception of 3D motion we consider the viewpoint relative to the motion; what follows are 18 cases of recurrent motion perception. To estimate repetition under all circumstances, our theory implies constructing a mixture of differential motion maps: mathbf {F}, {varvec{nabla }}mathbf {F}, {varvec{nabla }}{varvec{cdot }} mathbf {F} and {varvec{nabla }}{varvec{times }} mathbf {F}. We temporally convolve the motion maps with wavelet filters to estimate repetitive dynamics. Our method is able to spatially segment repetitive motion directly from the temporal filter responses densely computed over the motion maps. For experimental verification of our claims, we use our novel dataset for repetition estimation, better-reflecting reality with non-static and non-stationary repetitive motion. On the task of repetition counting, we obtain favorable results compared to a deep learning alternative.

Use of cues in virtual reality depends on visual feedback

3D motion perception is of central importance to daily life. However, when tested in laboratory settings, sensitivity to 3D motion signals is found to be poor, leading to the view that heuristics and prior assumptions are critical for 3D motion perception. Here we explore an alternative: sensitivity to 3D motion signals is context-dependent and must be learned based on explicit visual feedback in novel environments. The need for action-contingent visual feedback is well-established in the developmental literature. For example, young kittens that are passively moved through an environment, but unable to move through it themselves, fail to develop accurate depth perception. We find that these principles also obtain in adult human perception. Observers that do not experience visual consequences of their actions fail to develop accurate 3D motion perception in a virtual reality environment, even after prolonged exposure. By contrast, observers that experience the consequences of their actions improve performance based on available sensory cues to 3D motion. Specifically, we find that observers learn to exploit the small motion parallax cues provided by head jitter. Our findings advance understanding of human 3D motion processing and form a foundation for future study of perception in virtual and natural 3D environments.

The contributions of monocular and binocular signals to the perception of 3D motion

The contributions of monocular and binocular signals to the perception of 3D motion

Speed discrimination in the far monocular periphery: A relative advantage for interocular comparisons consistent with self-motion.

Some animals with lateral eyes (such as bees) control their navigation through the 3D world using velocity differences between the two eyes. Other animals with frontal eyes (such as primates, including humans) can perceive 3D motion based on the different velocities that a moving object projects upon the two retinae. Although one type of 3D motion perception involves a comparison between velocities from vastly different (monocular) portions of the visual field, and the other involves a comparison within overlapping (binocular) portions of the visual field, both compare velocities across the two eyes. Here we asked whether human interocular velocity comparisons, typically studied in the context of binocularly overlapping vision, operate in the far lateral (and hence, monocular) periphery and, if so, whether these comparisons were accordant with conventional interocular motion processing. We found that speed discrimination was indeed better between the two eyes' monocular visual fields, as compared to within a single eye's (monocular) visual field, but only when the velocities were consistent with commonly encountered motion. This intriguing finding suggests that mechanisms sensitive to relative motion information on opposite sides of an animal may have been retained, or at some point independently achieved, as the eyes became frontal in some animals.

Optimal Combination of the Binocular Cues to 3D Motion.

Perception necessarily entails combining separate sensory estimates into a single coherent whole. The perception of three-dimensional (3D) motion, for instance, can rely on two binocular cues: one related to the change in binocular disparity over time (CD) and the other related to interocular velocity differences (IOVD). Although previous work has shown that neither cue is strictly necessary for the perception of 3D motion, observers are able to judge 3D motion in displays in which one or the other cue has been eliminated, it is unclear whether or how the two cues are combined in situations in which both are present. We tested the visual performance of a sample of 81 individuals (Mage = 20.34, 49 females) in four main conditions that measured, respectively, static stereoacuity, CD, IOVD, and combined CD+IOVD sensitivity. We show that the sensitivity to the two binocular cues to 3D motion varies substantially across observers (CD: Md' = 1.01, SDd' = 1.1; IOVD: Md' = 1.16, SDd' = 1.03). Furthermore, sensitivity to the two cues was independent across observers (r[48] = 0.12, P = 0.42). Importantly, however, observed CD+IOVD performance was well-predicted based on the assumption that each observer combines the two cues in a statistically optimal fashion (r[79] = 0.75, P < 0.001). Our findings provide an explanation for the previously puzzling variability found in 3D perception across observers and laboratories, with some results suggesting that motion-in-depth percepts are largely determined by changes in binocular disparity, whereas others indicate that interocular velocity differences are key. Our results underline the existence of two complementary binocular mechanisms underlying 3D motion perception, with observers relying on these two mechanisms to different extents depending on their individual sensitivity.

Head tracking in virtual reality displays reduces the misperception of 3D motion

Observers rely on multiple complementary sensory cues to guide behavior. Previously, we showed that when observers estimate 3D object motion, they exhibit a surprising tendency to confuse approaching and receding motion (Fulvio et al, JOV 2014). However, such confusion seems to rarely occur in real world motion estimation. Extra-retinal cues such as head motion likely help disambiguate motion in depth. We tested this using the Oculus Rift, a head-mounted virtual reality display that can provide appropriate sensory feedback in response to changes in head orientation. We predicted that if observers rely on head motion to disambiguate motion in depth, accuracy in estimation of 3D object motion would improve with more natural viewing conditions. On each trial, a target with one of 3 contrast levels moved in a randomly chosen direction in the horizontal plane for 1s before disappearing. Observers (n=28) then adjusted the position of a ‘paddle’ around an invisible orbit so that it would intercept the target had it continued along the trajectory. Three head-motion conditions were tested: (i) ‘fixed’ – no updating with head orientation; (ii) ‘tracked’ – display updating with changes in head orientation; (iii) ‘lagged’ – display updating delayed by 50ms. Feedback was not provided. Reduced target contrast increased the proportion of motion in depth confusions across head-motion conditions (t(207)=-12.905, p< 0.001, d=0.89). Importantly, changes in head orientation were small (average max ~ 0.94deg in 3D space), yet had a significant effect on performance (t(207)=-3.3491, p< 0.001, d=0.23) - motion in depth confusions were less likely in the tracked versus fixed condition (t(166)=-1.94, p=0.027, d=0.42). We did not find differences between the tracked and lagged conditions (p=0.69), or fixed and lagged conditions (p=0.13). These results reveal a critical role for extra-retinal cues in 3D motion perception and contribute to the understanding of cross-modal integration. Meeting abstract presented at VSS 2015

Sensory uncertainty leads to systematic misperception of the direction of motion in depth.

Although we have made major advances in understanding motion perception based on the processing of lateral (2D) motion signals on computer displays, the majority of motion in the real (3D) world occurs outside of the plane of fixation, and motion directly toward or away from observers has particular behavioral relevance. Previous work has reported a systematic lateral bias in the perception of 3D motion, such that an object on a collision course with an observer's head is frequently judged to miss it, with obvious negative consequences. To better understand this bias, we systematically investigated the accuracy of 3D motion perception while manipulating sensory noise by varying the contrast of a moving target and its position in depth relative to fixation. Inconsistent with previous work, we found little bias under low sensory noise conditions. With increased sensory noise, however, we revealed a novel perceptual phenomenon: observers demonstrated a surprising tendency to confuse the direction of motion-in-depth, such that approaching objects were reported to be receding and vice versa. Subsequent analysis revealed that the lateral and motion-in-depth components of observers' reports are similarly affected, but that the effects on the motion-in-depth component (i.e., the motion-in-depth confusions) are much more apparent than those on the lateral component. In addition to revealing this novel visual phenomenon, these results shed new light on errors that can occur in motion perception and provide a basis for continued development of motion perception models. Finally, our findings suggest methods to evaluate the effectiveness of 3D visualization environments, such as 3D movies and virtual reality devices.

Systematic biases in 3D motion perception as a function of sensory uncertainty

Systematic biases in 3D motion perception as a function of sensory uncertainty

Systematic biases in 3D motion perception as a function of sensory uncertainty

1. Westheimer & Tanzman (1956) 8. Harris & Dean (2003) 2. Blakemore (1970) 9. Welchman et al. (2004) 3. Schumer & Julesz (1984) 10. Harris & Drga (2005) 4. Landers & Cormack (1997) 11. Lages (2006) 5. van Ee & Richards (2002) 12. Rushton & Duke (2007) 6. Thompson (1982) 13. Welchman et al. (2008) 7. Thompson & Stone (1982) 14. Duke & Rushton (2012) 3D motion perception is not inherently biased, but reduced contrast does increase variance At high contrast, 3D motion perception is largely accurate

Motion Perception and Driving: Predicting Performance Through Testing and Shortening Braking Reaction Times Through Training

A driving simulator was used to examine the relationship between motion perception and driving performance. Although motion perception test scores have been shown to be related to driving safety, it is not clear which combination of tests are the best predictors and whether motion perception training can improve driving performance. In experiment 1, 60 younger drivers (22.4 ± 2.5 years) completed three motion perception tests (2-dimensional [2D] motion-defined letter [MDL] identification, 3D motion in depth sensitivity [MID], and dynamic visual acuity [DVA]) followed by two driving tests (emergency braking [EB] and hazard perception [HP]). In experiment 2, 20 drivers (21.6 ± 2.1 years) completed 6 weeks of motion perception training (using the MDL, MID, and DVA tests), while 20 control drivers (22.0 ± 2.7 years) completed an online driving safety course. The EB performance was measured before and after training. In experiment 1, MDL (r = 0.34) and MID (r = 0.46) significantly correlated with EB score. The change in DVA score as a function of target speed (i.e., "velocity susceptibility") was correlated most strongly with HP score (r = -0.61). In experiment 2, the motion perception training group had a significant decrease in brake reaction time on the EB test from pre- to posttreatment, while there was no significant change for the control group: t(38) = 2.24, P = 0.03. Tests of 3D motion perception are the best predictor of EB, while DVA velocity susceptibility is the best predictor of hazard perception. Motion perception training appears to result in faster braking responses.

Aging of Low and High Level Vision: From Chromatic and Achromatic Contrast Sensitivity to Local and 3D Object Motion Perception

The influence of normal aging in early, intermediate and high-level visual processing is still poorly understood. We have addressed this important issue in a large cohort of 653 subjects divided into five distinct age groups, [20;30[, [30;40[, [40;50[, [50;60[and [60;[. We applied a broad range of psychophysical tests, testing distinct levels of the visual hierarchy, from local processing to global integration, using simple gratings (spatial contrast sensitivity -CS- using high temporal/low spatial frequency or intermediate spatial frequency static gratings), color CS using Landolt patches, moving dot stimuli (Local Speed Discrimination) and dot patterns defining 3D objects (3D Structure from Motion, 3D SFM). Aging data were fitted with linear or quadratic regression models, using the adjusted coefficient of determination (R2 a) to quantify the effect of aging. A significant effect of age was found on all visual channels tested, except for the red-green chromatic channel. The high temporal low spatial frequency contrast sensitivity channel showed a mean sensitivity loss of 0.75 dB per decade (R2 a = 0.17, p<0.001), while the lower intermediate spatial frequency channel showed a more pronounced decrease, around 2.35 dB (R2 a = 0.55, p<0.001). Concerning low-level motion perception, speed discrimination decreased 2.71°/s (R2 a = 0.18, p<0.001) and 3.15°/s (R2 a = 0.13, p<0.001) only for short presentations for horizontal and oblique meridians, respectively. The 3D SFM task, requiring high-level integration across dorsal and ventral streams, showed the strongest (quadratic) decrease of motion coherence perception with age, especially when the task was temporally constrained (R2 a = 0.54, p<0.001). These findings show that visual channels are influenced by aging into different extent, with time presenting a critical role, and high-level dorso-ventral dominance of deterioration, which accelerates with aging, in contrast to the other channels that show a linear pattern of deterioration.

Straight or curved? From deterministic to probabilistic models of 3D motion perception

Straight or curved? From deterministic to probabilistic models of 3D motion perception

Low-level motion analysis of color and luminance for perception of 2D and 3D motion

We investigated the low-level motion mechanisms for color and luminance and their integration process using 2D and 3D motion aftereffects (MAEs). The 2D and 3D MAEs obtained in equiluminant color gratings showed that the visual system has the low-level motion mechanism for color motion as well as for luminance motion. The 3D MAE is an MAE for motion in depth after monocular motion adaptation. Apparent 3D motion can be perceived after prolonged exposure of one eye to lateral motion because the difference in motion signal between the adapted and unadapted eyes generates interocular velocity differences (IOVDs). Since IOVDs cannot be analyzed by the high-level motion mechanism of feature tracking, we conclude that a low-level motion mechanism is responsible for the 3D MAE. Since we found different temporal frequency characteristics between the color and luminance stimuli, MAEs in the equiluminant color stimuli cannot be attributed to a residual luminance component in the color stimulus. Although a similar MAE was found with a luminance and a color test both for 2D and 3D motion judgments after adapting to either color or luminance motion, temporal frequency characteristics were different between the color and luminance adaptation. The visual system must have a low-level motion mechanism for color signals as for luminance ones. We also found that color and luminance motion signals are integrated monocularly before IOVD analysis, showing a cross adaptation effect between color and luminance stimuli. This was supported by an experiment with dichoptic presentations of color and luminance tests. In the experiment, color and luminance tests were presented in the different eyes dichoptically with four different combinations of test and adaptation: color or luminance test in the adapted eye after color or luminance adaptation. Findings of little or no influence of the adaptation/test combinations indicate the integration of color and luminance motion signals prior to the binocular IOVD process.

Motion processing with two eyes in three dimensions

The movement of an object toward or away from the head is perhaps the most critical piece of information an organism can extract from its environment. Such 3D motion produces horizontally opposite motions on the two retinae. Little is known about how or where the visual system combines these two retinal motion signals, relative to the wealth of knowledge about the neural hierarchies involved in 2D motion processing and binocular vision. Canonical conceptions of primate visual processing assert that neurons early in the visual system combine monocular inputs into a single cyclopean stream (lacking eye-of-origin information) and extract 1D ("component") motions; later stages then extract 2D pattern motion from the cyclopean output of the earlier stage. Here, however, we show that 3D motion perception is in fact affected by the comparison of opposite 2D pattern motions between the two eyes. Three-dimensional motion sensitivity depends systematically on pattern motion direction when dichoptically viewing gratings and plaids-and a novel "dichoptic pseudoplaid" stimulus provides strong support for use of interocular pattern motion differences by precluding potential contributions from conventional disparity-based mechanisms. These results imply the existence of eye-of-origin information in later stages of motion processing and therefore motivate the incorporation of such eye-specific pattern-motion signals in models of motion processing and binocular integration.