Retinal Motion Patterns Research Articles

Award Lecture: 2022 Tillyer Award Lecture: Visual Control of Locomotion in Natural Environments.

Control of locomotion is clearly an important role for vision, but one that we know little about. Considerable attention has been devoted to the potential role of retinal motion patterns, generated by self-motion, in steering towards a goal. These motion patterns are determined by an interconnected set of factors, including gaze location, gaze stabilization, the structure of the environment, and the walker's behavioral goals. We have collected measurements of the eyes, the body, and the 3D environment during locomotion in natural terrains. This has allowed us to explore the resulting retinal motion patterns, and how they might be used to guide behavior. Large variations in head direction during the gait cycle make it problematic for use in steering towards a goal, but gaze stabilization leads to regularities in the retinal motion patterns that could provide a postural control signal during locomotion. Another aspect of the visual control of locomotion is the choice of foot placement in rugged terrain. We examined the role of local height changes in the terrain, and showed that visual selection of footholds reflects a complex interplay between energetic costs and the need for stable foot placement. Thus visual signals about motion and depth act as time-varying local control signals for the momentary disposition of the body rather than steering toward more distant goals.

Retinal motion statistics during natural locomotion.

Walking through an environment generates retinal motion, which humans rely on to perform a variety of visual tasks. Retinal motion patterns are determined by an interconnected set of factors, including gaze location, gaze stabilization, the structure of the environment, and the walker's goals. The characteristics of these motion signals have important consequences for neural organization and behavior. However, to date, there are no empirical in situ measurements of how combined eye and body movements interact with real 3D environments to shape the statistics of retinal motion signals. Here, we collect measurements of the eyes, the body, and the 3D environment during locomotion. We describe properties of the resulting retinal motion patterns. We explain how these patterns are shaped by gaze location in the world, as well as by behavior, and how they may provide a template for the way motion sensitivity and receptive field properties vary across the visual field.

Self-motion illusions from distorted optic flow in multifocal glasses

SummaryProgressive addition lenses (PALs) are ophthalmic lenses to correct presbyopia by providing improvements of near and far vision in different areas of the lens, but distorting the periphery of the wearer's field of view. Distortion-related difficulties reported by PAL wearers include unnatural self-motion perception. Visual self-motion perception is guided by optic flow, the pattern of retinal motion produced by self-motion. We tested the influence of PAL distortions on optic flow-based heading estimation using a model of heading perception and a virtual reality-based psychophysical experiment. The model predicted changes of heading estimation along a vertical axis, depending on visual field size and gaze direction. Consistent with this prediction, participants experienced upwards deviations of self-motion when gaze through the periphery of the lens was simulated, but not for gaze through the center. We conclude that PALs may lead to illusions of self-motion which could be remedied by a careful gaze strategy.

A model of how depth facilitates scene-relative object motion perception.

Many everyday interactions with moving objects benefit from an accurate perception of their movement. Self-motion, however, complicates object motion perception because it generates a global pattern of motion on the observer’s retina and radically influences an object’s retinal motion. There is strong evidence that the brain compensates by suppressing the retinal motion due to self-motion, however, this requires estimates of depth relative to the object—otherwise the appropriate self-motion component to remove cannot be determined. The underlying neural mechanisms are unknown, but neurons in brain areas MT and MST may contribute given their sensitivity to motion parallax and depth through joint direction, speed, and disparity tuning. We developed a neural model to investigate whether cells in areas MT and MST with well-established neurophysiological properties can account for human object motion judgments during self-motion. We tested the model by comparing simulated object motion signals to human object motion judgments in environments with monocular, binocular, and ambiguous depth. Our simulations show how precise depth information, such as that from binocular disparity, may improve estimates of the retinal motion pattern due the self-motion through increased selectivity among units that respond to the global self-motion pattern. The enhanced self-motion estimates emerged from recurrent feedback connections in MST and allowed the model to better suppress the appropriate direction, speed, and disparity signals from the object’s retinal motion, improving the accuracy of the object’s movement direction represented by motion signals.

Motion integration is anisotropic during smooth pursuit eye movements.

Smooth pursuit eye movements (pursuit) are used to minimize the retinal motion of moving objects. During pursuit, the pattern of motion on the retina carries not only information about the object movement but also reafferent information about the eye movement itself. The latter arises from the retinal flow of the stationary world in the direction opposite to the eye movement. To extract the global direction of motion of the tracked object and stationary world, the visual system needs to integrate ambiguous local motion measurements (i.e., the aperture problem). Unlike the tracked object, the stationary world's global motion is entirely determined by the eye movement and thus can be approximately derived from motor commands sent to the eye (i.e., from an efference copy). Because retinal motion opposite to the eye movement is dominant during pursuit, different motion integration mechanisms might be used for retinal motion in the same direction and opposite to pursuit. To investigate motion integration during pursuit, we tested direction discrimination of a brief change in global object motion. The global motion stimulus was a circular array of small static apertures within which one-dimensional gratings moved. We found increased coherence thresholds and a qualitatively different reflexive ocular tracking for global motion opposite to pursuit. Both effects suggest reduced sampling of motion opposite to pursuit, which results in an impaired ability to extract coherence in motion signals in the reafferent direction. We suggest that anisotropic motion integration is an adaptation to asymmetric retinal motion patterns experienced during pursuit eye movements. NEW & NOTEWORTHY This study provides a new understanding of how the visual system achieves coherent perception of an object's motion while the eyes themselves are moving. The visual system integrates local motion measurements to create a coherent percept of object motion. An analysis of perceptual judgments and reflexive eye movements to a brief change in an object's global motion confirms that the visual and oculomotor systems pick fewer samples to extract global motion opposite to the eye movement.

Viewpoint oscillation improves the perception of distance travelled based on optic flow.

When static observers are presented with a visual simulation of forward self-motion, they generally misestimate distance travelled relative to a previously seen distant target: It has been suggested that this finding can be accounted for by a "leaky path integration" model. In the present study, using a similar experimental procedure, this result was confirmed. It was also established that combining the translational optical flow with simulated head oscillations (similar to those during natural walking) improved the subjects' perception of the distance travelled in comparison with a purely translational flow. This improvement may be attributable to the fact that an optic flow pattern resembling that associated with walking enhances the path integration process. In a subsequent experiment, we investigated whether it was the biological or the rhythmical characteristics of the simulation that enhanced the subjects' estimates of the distance travelled. The results obtained confirm that adding rhythmic components to the optic flow pattern improved the accuracy of subjects' perception of the distance travelled. However, no significant differences between biological and rhythmical oscillations were detected. These results relate to recent studies on the effects of smooth and jittering optic flows on vection onset and strength. One possible conclusion is that oscillations may increase the global retinal motion and thus improve the vection and path integration processes. Another possibility is that the nonmonotonous pattern of retinal motion induced by oscillatory inputs may maintain optimum sensitivity to the optic flow over time and thus improve the accuracy of subjects' perception of the distance travelled.

OSCILLATORY OPTICAL FLOWS IMPROVE THE PERCEPTION OF TRAVELLED DISTANCE IN STATIC OBSERVERS

When static observers are exposed to a visual simulation of forward self-motion, they generally overestimate travelled distance relative to a previously seen distant target (Redlick, Harris, & Jenkin, 2001). Using the same experimental paradigm, we reproduced this result, which can be accounted for by a leaky path integration model (Lappe, Jenkin, & Harris, 2007). However, we also showed that a translational optical flow with additional oscillatory components, simulating head motion during walking, allows for better perception of travelled distance, as compared to a purely translational flow. This result can be discussed as an improvement of path integration, using an ecological optic flow pattern (close to natural walking feedback). In a subsequent experiment, we tested if it is the biological or the rhythmical nature of the simulation which provides better estimates of travelled distance. Participants were standing and passively confronted to three conditions of virtual simulation of self-motion in a 4-sided CAVE: (1) a translational optical flow (2) a translational flow, with an added component, via triangular oscillations in the vertical axis and (3) a flow, reproducing motion of the head during walking, as previously measured and modelled using a motion capture system. The results confirm that the addition of rhythmical components in the optic flow pattern improve the accuracy of travelled distance. However, they failed to reveal a significant difference between and rhythmic oscillations. Further experiments on the effect of spatio-temporal oscillatory components in the optical flow are required. However, these results can be related to recent studies comparing the effect of smooth and jittering optic flows on vection onset and strength (Palmisano et al., 2014). One suggestion is that oscillatory components produce a non-monotonous pattern of retinal motion, maintaining optimal sensitivity to optic flow, and consequently improving the perception of travelled distance. Meeting abstract presented at VSS 2016

During self-movement humans are better at judging whether an object is moving (flow parsing) than whether they will hit it (heading).

During locomotion we can use information in the retinal flow field to judge whether we will pass to the left or right of an object in the scene (heading). We can also use information in retinal flow to judge whether an object is moving relative to the scene (flow parsing). Both judgements rely on the brain identifying optic flow (global patterns of retinal motion that are characteristic of self-movement). How does the precision of these two judgments compare? Differences or similarities in precision may provide some insight into the underpinning mechanisms. We designed stimuli that allowed direct comparison of the precision of the two judgements. In the heading task, we simulated forward movement of the observer by moving a rigid scene containing wireframe objects, plus a coloured target sphere, towards the observer. Observers judged whether they would pass to the left or right of the sphere. Thresholds were measured as a function of the angle between the sphere and the simulated direction of movement. In the object movement task, the background scene and the target sphere moved on slightly different trajectories which led to a perception of scene-relative object movement. Observers judged whether the sphere was moving to the left or right in the scene. Thresholds were measured a function of the difference in direction of the background and target trajectories. Precision was significantly higher when judging object movement than when judging self-movement. Control experiments removed local cues and confirmed that the results were unchanged. In a simulated rotation experiment we found that performance on the object movement task was unaffected while judgements of self-movement were impaired. These results are compatible with the hypothesis that flow parsing is not dependent upon a prior estimate of heading; flow-parsing is either an independent process, or it precedes estimates of heading. Meeting abstract presented at VSS 2015. (Less)

Motion as a source of environmental information: a fresh view on biological motion computation by insect brains.

Despite their miniature brains insects, such as flies, bees and wasps, are able to navigate by highly erobatic flight maneuvers in cluttered environments. They rely on spatial information that is contained in the retinal motion patterns induced on the eyes while moving around (“optic flow”) to accomplish their extraordinary performance. Thereby, they employ an active flight and gaze strategy that separates rapid saccade-like turns from translatory flight phases where the gaze direction is kept largely constant. This behavioral strategy facilitates the processing of environmental information, because information about the distance of the animal to objects in the environment is only contained in the optic flow generated by translatory motion. However, motion detectors as are widespread in biological systems do not represent veridically the velocity of the optic flow vectors, but also reflect textural information about the environment. This characteristic has often been regarded as a limitation of a biological motion detection mechanism. In contrast, we conclude from analyses challenging insect movement detectors with image flow as generated during translatory locomotion through cluttered natural environments that this mechanism represents the contours of nearby objects. Contrast borders are a main carrier of functionally relevant object information in artificial and natural sceneries. The motion detection system thus segregates in a computationally parsimonious way the environment into behaviorally relevant nearby objects and—in many behavioral contexts—less relevant distant structures. Hence, by making use of an active flight and gaze strategy, insects are capable of performing extraordinarily well even with a computationally simple motion detection mechanism.

A new spin on vection in depth.

Previous research has shown that adding lateral viewpoint changes to visual displays simulating self-motion in depth can increase the strength of linear vection. We performed experiments to determine whether these vection increases are caused by reduced adaptation to retinal motion, rather than increased motion parallax in the visual display. In Experiment 1, we added increasing amplitudes of sinusoidal angular viewpoint oscillation around the viewing axis (up to 94.2°/s) to radial flow simulating self-motion in depth. We found that angular viewpoint oscillation systematically reduced the onset latencies and increased the overall strength of vection in depth, compared with pure radial flow. In Experiment 2, we compared vection strength between radial flow displays with either added angular oscillation or continuous spiral rotation of equivalent peak velocity around the viewing axis (62.8°/s), and found that angular viewpoint oscillation generated the strongest vection. In Experiment 3, we found that pure radial flow with or without continuous spiral rotation produced radial motion aftereffects that lasted longer than that produced by radial flow with angular viewpoint oscillation. These findings support the view that the way viewpoint oscillation increases vection does not critically depend on motion parallax, but rather, on a changing pattern of retinal motion that serves to reduce visual adaptation and sustain sensitivity to optic flow.

Spared ability to perceive direction of locomotor heading and scene-relative object movement despite inability to perceive relative motion.

BackgroundAll contemporary models of perception of locomotor heading from optic flow (the characteristic patterns of retinal motion that result from self-movement) begin with relative motion. Therefore it would be expected that an impairment on perception of relative motion should impact on the ability to judge heading and other 3D motion tasks.Material/MethodsWe report two patients with occipital lobe lesions whom we tested on a battery of motion tasks. Patients were impaired on all tests that involved relative motion in plane (motion discontinuity, form from differences in motion direction or speed). Despite this they retained the ability to judge their direction of heading relative to a target. A potential confound is that observers can derive information about heading from scale changes bypassing the need to use optic flow. Therefore we ran further experiments in which we isolated optic flow and scale change.ResultsPatients’ performance was in normal ranges on both tests. The finding that ability to perceive heading can be retained despite an impairment on ability to judge relative motion questions the assumption that heading perception proceeds from initial processing of relative motion. Furthermore, on a collision detection task, SS and SR’s performance was significantly better for simulated forward movement of the observer in the 3D scene, than for the static observer. This suggests that in spite of severe deficits on relative motion in the frontoparlel (xy) plane, information from self-motion helped identification objects moving along an intercept 3D relative motion trajectory.ConclusionsThis result suggests a potential use of a flow parsing strategy to detect in a 3D world the trajectory of moving objects when the observer is moving forward. These results have implications for developing rehabilitation strategies for deficits in visually guided navigation.

Detecting object movement during self-movement: The importance of local motion contrast, position change and optic flow

We have shown that observers can identify object movement within the complex pattern of retinal motion that arises during movement of the observer. We hypothesised (Rushton & Warren, 2005) that this process exploits the brain's well-documented sensitivity to optic flow (global patterns of motion that are characteristic of self-movement) and data collected in previous studies is compatible with this hypothesis. Here we report strong direct tests of alternative solutions. We first remove the retinal disparities within the scene that are necessary for object movement to be identified by local (within disparity plane) motion contrast; we do this by presenting the scene to a single eye, whilst presenting the probe object (which varies in depth) to both eyes. We find the observers still demonstrate behaviour compatible with the identification of object movement. In a second study, we investigate the effect of removing motion (flow) from the display. We do this by presenting either a natural continuous change in (simulated) observer viewpoint, or an unnatural step change. We find that in the latter case, despite local motion and position change information being preserved, performance is significantly impaired. Taken together these results point to a central role for optic flow processing in the identification of scene-relative object movement.

Optic Flow Processing for the Assessment of Object Movement during Ego Movement

The vast majority of research on optic flow (retinal motion arising because of observer movement) has focused on its use in heading recovery and guidance of locomotion. Here we demonstrate that optic flow processing has an important role in the detection and estimation of scene-relative object movement during self movement. To do this, the brain identifies and globally discounts (i.e., subtracts) optic flow patterns across the visual scene-a process called flow parsing. Remaining motion can then be attributed to other objects in the scene. In two experiments, stationary observers viewed radial expansion flow fields and a moving probe at various onscreen locations. Consistent with global discounting, perceived probe motion had a significant component toward the center of the display and the magnitude of this component increased with probe eccentricity. The contribution of local motion processing to this effect was small compared to that of global processing (experiment 1). Furthermore, global discounting was clearly implicated because these effects persisted even when all the flow in the hemifield containing the probe was removed (experiment 2). Global processing of optic flow information is shown to play a fundamental role in the recovery of object movement during ego movement.

Evidence for flow-parsing in radial flow displays

Retinal motion of objects is not in itself enough to signal whether or how objects are moving in the world; the same pattern of retinal motion can result from movement of the object, the observer or both. Estimation of scene-relative movement of an object is vital for successful completion of many simple everyday tasks. Recent research has provided evidence for a neural flow-parsing mechanism which uses the brain’s sensitivity to optic flow to separate retinal motion signals into those components due to observer movement and those due to the movement of objects in the scene. In this study we provide further evidence that flow-parsing is implicated in the assessment of object trajectory during observer movement. Furthermore, it is shown that flow-parsing involves a global analysis of retinal motion, as might be expected if optic flow processing underpinned this mechanism.

The use of direction and distance information in the perception of approach trajectory

A pair of projectiles travelling on parallel trajectories produce differing patterns of retinal motion when they originate at different distances. For an observer to recognise that the two trajectories are parallel she must “factor out” the effect of distance on retinal motion. The observer faces a similar problem when physically parallel trajectories originate at different lateral positions; here direction must be “factored out”. We report the results of a series of experiments designed to determine if observers can do this. The observers’ task was to judge whether the direction of travel of an approaching sphere (test trajectory) was to the left or right of parallel to a previously shown trajectory (reference trajectory). In the first set of experiments the reference and test trajectories started from different lateral positions. In the final experiment they started from different distances. From the pattern of judgements we determined a set of perceptually parallel trajectories. Perceptually parallel trajectories deviated significantly from physically parallel. We conclude that under circumstances comparable to those encountered when catching a ball in flight, observers do not have access to accurate estimates of trajectory direction.

The pop out of scene-relative object movement against retinal motion due to self-movement

An object that moves is spotted almost effortlessly; it “pops out”. When the observer is stationary, a moving object is uniquely identified by retinal motion. This is not so when the observer is also moving; as the eye travels through space all scene objects change position relative to the eye producing a complicated field of retinal motion. Without the unique identifier of retinal motion an object moving relative to the scene should be difficult to locate. Using a search task, we investigated this proposition. Computer-rendered objects were moved and transformed in a manner consistent with movement of the observer. Despite the complex pattern of retinal motion, objects moving relative to the scene were found to pop out. We suggest the brain uses its sensitivity to optic flow to “stabilise” the scene, allowing the scene-relative movement of an object to be identified.

Moving observers, relative retinal motion and the detection of object movement

Motion of the image of an object across the retina (or a camera sensor) may be due to movement of the object, movement of the observation point or a combination of the two. Humans are able to routinely distinguish between these causes and correctly perceive whether an object of interest is in motion or scene-stationary. The important question is how this ubiquitous and difficult problem is solved. We have investigated whether the brain can resolve the ambiguity by comparing the retinal motion of the object of interest — the target — to that of scene objects.

Visual motion due to eye movements helps guide the hand

Movement of the body, head, or eyes with respect to the world creates one of the most common yet complex situations in which the visuomotor system must localize objects. In this situation, vestibular, proprioceptive, and extra-retinal information contribute to accurate visuomotor control. The utility of retinal motion information, on the other hand, is questionable, since a single pattern of retinal motion can be produced by any number of head or eye movements. Here we investigated whether retinal motion during a smooth pursuit eye movement contributes to visuomotor control. When subjects pursued a moving object with their eyes and reached to the remembered location of a separate stationary target, the presence of a moving background significantly altered the endpoints of their reaching movements. A background that moved with the pursuit, creating a retinally stationary image (no retinal slip), caused the endpoints of the reaching movements to deviate in the direction of pursuit, overshooting the target. A physically stationary background pattern, however, producing retinal image motion opposite to the direction of pursuit, caused reaching movements to become more accurate. The results indicate that background retinal motion is used by the visuomotor system in the control of visually guided action.

SENSORIMOTOR CONVERGENCE IN VISUAL NAVIGATION AND FLIGHT CONTROL SYSTEMS

Insects exhibit unparalleled and incredibly robust flight dynamics in the face of uncertainties. A fundamental principle contributing to this amazing behavior is rapid processing and convergence of visual sensory information to flight motor commands via spatial wide-field integration, accomplished by motion pattern sensitive interneurons in the lobula plate portion of the visual ganglia. Within a control-theoretic framework, a model for wide-field integration of retinal image flow is developed, establishing the connection between image flow kernels (retinal motion pattern sensitivities) and the feedback terms they represent. It is demonstrated that the proposed output feedback methodology is sufficient to give rise to experimentally observed navigational heuristics as the centering and forward speed regulation responses exhibited by honeybees.

Pursuit compensation during self-motion.

The pattern of motion in the retinal image during self-motion contains information about the person's movement. Pursuit eye movements perturb the pattern of retinal-image motion, complicating the problem of self-motion perception. A question of considerable current interest is the relative importance of retinal and extra-retinal signals in compensating for these effects of pursuit on the retinal image. We addressed this question by examining the effect of prior motion stimuli on self-motion judgments during pursuit. Observers viewed 300 ms random-dot displays simulating forward self-motion during pursuit to the right or to the left; at the end of each display a probe appeared and observers judged whether they would pass left or right of it. The display was preceded by a 300 ms dot pattern that was either stationary or moved in the same direction as, or opposite to, the eye movement. This prior motion stimulus had a large effect on self-motion judgments when the simulated scene was a frontoparallel wall (experiment 1), but not when it was a three-dimensional (3-D) scene (experiment 2). Corresponding simulated-pursuit conditions controlled for purely retinal motion aftereffects, implying that the effect in experiment 1 is mediated by an interaction between retinal and extra-retinal signals. In experiment 3, we examined self-motion judgments with respect to a 3-D scene with mixtures of real and simulated pursuit. When real and simulated pursuits were in opposite directions, performance was determined by the total amount of pursuit-related retinal motion, consistent with an extra-retinal 'trigger' signal that facilitates the action of a retinally based pursuit-compensation mechanism. However, results of experiment 1 without a prior motion stimulus imply that extra-retinal signals are more informative when retinal information is lacking. We conclude that the relative importance of retinal and extra-retinal signals for pursuit compensation varies with the informativeness of the retinal motion pattern, at least for short durations. Our results provide partial explanations for a number of findings in the literature on perception of self-motion and motion in the frontal plane.