Translational Optic Flow Research Articles

Moving object detection based on bioinspired background subtraction

Flying insects rely mainly upon visual motion to detect and track objects. There has been a lot of research on fly inspired algorithms for object detection, but few have been developed based on visual motion alone. One of the daunting difficulties is that the neural and circuit mechanisms underlying the foreground-background segmentation are still unclear. Our previous modeling study proposed that the lobula held parallel pathways with distinct directional selectivity, each of which could retinotopically discriminate figures moving in its own preferred direction based on relative motion cues. The previous model, however, did not address how the multiple parallel pathways gave the only detection output at their common downstream. Since the preferred directions of the pathways along either horizontal or vertical axis were opposite to each other, the background moving in the opposite direction to an object also activated the corresponding lobula pathway. Indiscriminate or ungated projection from all the pathways to their downstream would mix objects with the moving background, making the previous model fail with non-stationary background. Here, we extend the previous model by proposing that the background motion-dependent gating of individual lobula projections is the key to object detection. Large-field lobula plate tangential cells are hypothesized to perform the gating to realize bioinspired background subtraction. The model is shown to be capable of implementing a robust detection of moving objects in video sequences with either a moving camera that induces translational optic flow or a static camera. The model sheds light on the potential of the concise fly algorithm in real-world applications.

Parallel motion vision pathways in the brain of a tropical bee

Spatial orientation is a prerequisite for most behaviors. In insects, the underlying neural computations take place in the central complex (CX), the brain’s navigational center. In this region different streams of sensory information converge to enable context-dependent navigational decisions. Accordingly, a variety of CX input neurons deliver information about different navigation-relevant cues. In bees, direction encoding polarized light signals converge with translational optic flow signals that are suited to encode the flight speed of the animals. The continuous integration of speed and directions in the CX can be used to generate a vector memory of the bee’s current position in space in relation to its nest, i.e., perform path integration. This process depends on specific, complex features of the optic flow encoding CX input neurons, but it is unknown how this information is derived from the visual periphery. Here, we thus aimed at gaining insight into how simple motion signals are reshaped upstream of the speed encoding CX input neurons to generate their complex features. Using electrophysiology and anatomical analyses of the halictic bees Megalopta genalis and Megalopta centralis, we identified a wide range of motion-sensitive neurons connecting the optic lobes with the central brain. While most neurons formed pathways with characteristics incompatible with CX speed neurons, we showed that one group of lobula projection neurons possess some physiological and anatomical features required to generate the visual responses of CX optic-flow encoding neurons. However, as these neurons cannot explain all features of CX speed cells, local interneurons of the central brain or alternative input cells from the optic lobe are additionally required to construct inputs with sufficient complexity to deliver speed signals suited for path integration in bees.

Indoor and outdoor in-flight odometry based solely on optic flows with oscillatory trajectories

Estimating distance traveled is a frequently arising problem in robotic applications designed for use in environments where GPS is only intermittently or not at all available. In UAVs, the presence of weight and computational power constraints makes it necessary to develop odometric strategies based on minimilastic equipment. In this study, a hexarotor was made to perform up-and-down oscillatory movements while flying forward in order to test a self-scaled optic flow based odometer. The resulting self-oscillatory trajectory generated series of contractions and expansions in the optic flow vector field, from which the flight height of the hexarotor could be estimated using an Extended Kalman Filter. For the odometry, the downward translational optic flow was scaled by this current visually estimated flight height before being mathematically integrated to obtain the distance traveled. Here we present three strategies based on sensor fusion requiring no, precise or rough prior knowledge of the optic flow variations generated by the sinusoidal trajectory. The “rough prior knowledge” strategy is based on the shape and timing of the variations in the optic flow. Tests were performed first in a flight arena, where the hexarotor followed a circular trajectory while oscillating up and down over a distance of about [Formula: see text] m under illuminances of [Formula: see text] lux and [Formula: see text] lux. Preliminary field tests were then performed, in which the hexarotor followed a longitudinal bouncing [Formula: see text]-long trajectory over an irregular pattern of grass.

Contrast independent biologically inspired translational optic flow estimation

The visual systems of insects are relatively simple compared to humans. However, they enable navigation through complex environments where insects perform exceptional levels of obstacle avoidance. Biology uses two separable modes of optic flow to achieve this: rapid gaze fixation (rotational motion known as saccades); and the inter-saccadic translational motion. While the fundamental process of insect optic flow has been known since the 1950’s, so too has its dependence on contrast. The surrounding visual pathways used to overcome environmental dependencies are less well known. Previous work has shown promise for low-speed rotational motion estimation, but a gap remained in the estimation of translational motion, in particular the estimation of the time to impact. To consistently estimate the time to impact during inter-saccadic translatory motion, the fundamental limitation of contrast dependence must be overcome. By adapting an elaborated rotational velocity estimator from literature to work for translational motion, this paper proposes a novel algorithm for overcoming the contrast dependence of time to impact estimation using nonlinear spatio-temporal feedforward filtering. By applying bioinspired processes, approximately 15 points per decade of statistical discrimination were achieved when estimating the time to impact to a target across 360 background, distance, and velocity combinations: a 17-fold increase over the fundamental process. These results show the contrast dependence of time to impact estimation can be overcome in a biologically plausible manner. This, combined with previous results for low-speed rotational motion estimation, allows for contrast invariant computational models designed on the principles found in the biological visual system, paving the way for future visually guided systems.

Spatial tuning of translational optic flow responses in hawkmoths of varying body size

To safely navigate their environment, flying insects rely on visual cues, such as optic flow. Which cues insects can extract from their environment depends closely on the spatial and temporal response properties of their visual system. These in turn can vary between individuals that differ in body size. How optic flow-based flight control depends on the spatial structure of visual cues, and how this relationship scales with body size, has previously been investigated in insects with apposition compound eyes. Here, we characterised the visual flight control response limits and their relationship to body size in an insect with superposition compound eyes: the hummingbird hawkmoth Macroglossum stellatarum. We used the hawkmoths’ centring response in a flight tunnel as a readout for their reception of translational optic flow stimuli of different spatial frequencies. We show that their responses cut off at different spatial frequencies when translational optic flow was presented on either one, or both tunnel walls. Combined with differences in flight speed, this suggests that their flight control was primarily limited by their temporal rather than spatial resolution. We also observed strong individual differences in flight performance, but no correlation between the spatial response cutoffs and body or eye size.

Oscillations make a self-scaled model for honeybees' visual odometer reliable regardless of flight trajectory.

Honeybees foraging and recruiting nest-mates by performing the waggle dance need to be able to gauge the flight distance to the food source regardless of the wind and terrain conditions. Previous authors have hypothesized that the foragers’ visual odometer mathematically integrates the angular velocity of the ground image sweeping backward across their ventral viewfield, known as translational optic flow. The question arises as to how mathematical integration of optic flow (usually expressed in radians/s) can reliably encode distances, regardless of the height and speed of flight. The vertical self-oscillatory movements observed in honeybees trigger expansions and contractions of the optic flow vector field, yielding an additional visual cue called optic flow divergence. We have developed a self-scaled model for the visual odometer in which the translational optic flow is scaled by the visually estimated current clearance from the ground. In simulation, this model, which we have called SOFIa, was found to be reliable in a large range of flight trajectories, terrains and wind conditions. It reduced the statistical dispersion of the estimated flight distances approximately 10-fold in comparison with the mathematically integrated raw optic flow model. The SOFIa model can be directly implemented in robotic applications based on minimalistic visual equipment.

Quadrotor going through a window and landing: An image-based visual servo control approach

This paper considers the problem of controlling a quadrotor to go through a window and land on a planar target, the landing pad, using an Image-Based Visual Servo (IBVS) controller that relies on sensing information from two on-board cameras and an IMU. The maneuver is divided into two stages: crossing the window and landing on the pad. For the first stage, a control law is proposed that guarantees that the vehicle will not collide with the wall containing the window and will go through the window with non-zero velocity along the direction orthogonal to the window, keeping at all times a safety distance with respect to the window edges. For the landing stage, the proposed control law ensures that the vehicle achieves a smooth touchdown, keeping at all time a positive height above the plane containing the landing pad. For control purposes, the centroid vectors provided by the combination of the spherical image measurements of a collection of landmarks (corners) for both the window and the landing pad are used as position measurement. The translational optical flow relative to the wall, window edges, and landing plane is used as velocity cue. To achieve the proposed objective, no direct measurements nor explicit estimate of position or velocity are required. Simulation and experimental results are provided to illustrate the performance of the presented controller.

Stabilizing responses to sideslip disturbances in Drosophila melanogaster are modulated by the density of moving elements on the ground.

Stabilizing responses to sideslip disturbances are a critical part of the flight control system in flies. While strongly mediated by mechanoreception, much of the final response results from the wide-field motion detection system associated with vision. In order to be effective, these responses must match the disturbance they are aimed to correct. To do this, flies must estimate the velocity of the disturbance, although it is not known how they accomplish this task when presented with natural images or dot fields. The recent finding, that motion parallax in dot fields can modulate stabilizing responses only if perceived below the fly, raises the question of whether other image statistics are also processed differently between eye regions. One such parameter is the density of elements moving in translational optic flow. Depending on the habitat, there might be strong differences in the density of elements providing information about self-motion above and below the fly, which in turn could act as selective pressures tuning the visual system to process this parameter on a regional basis. By presenting laterally moving dot fields of different densities we found that, in Drosophila melanogaster, the amplitude of the stabilizing response is significantly affected by the number of elements in the field of view. Flies countersteer strongly within a relatively low and narrow range of element densities. But this effect is exclusive to the ventral region of the eye, and dorsal stimuli elicit an unaltered and stereotypical response regardless of the density of elements in the flow. This highlights local specialization of the eye and suggests the lower region may play a more critical role in translational flight stabilization.

Sudden visual perturbations induce postural responses in a virtual reality environment

Virtual reality environments can be manipulated allowing controlled visual disturbances. These environmental manipulations (textures, objects, and movements) can affect postural control by creating a sensory mismatch, making it possible to investigate multisensory reweighting. VR integration with motion capture equipment is relatively new; these environmental factors have not received sufficient attention to be employed in postural related research and in VR environment development. Seventeen subjects performed 48 trials of forward and backward translational optic flow in single and double support presented on a head mounted display. Each condition was presented in three different velocities: 3 m/s, 5 m/s and 8 m/s. The effect on center of pressure distance and velocity were analyzed. A repeated measures, counterbalanced experimental design was used. Center of pressure distance travelled increased in the single support condition (p < 0.001). The largest single support postural disturbance was observed in the slowest velocity and longest duration (3 m/s) and the smallest in the fastest velocity shortest duration (8 m/s). A slow optic flow velocity and higher duration of sudden visual translations generates greater disruption to postural control in single support. The velocity and duration of perturbations need to be carefully considered when studying human postural control and the design of VR environments.

Static object imaging features recognition algorithm in dynamic scene mapping

In dynamic scene monocular visual SLAM, it is important to recognize static object imaging features for mapping. The optical flow of noise and the object moving in the same orientations would have the same orientations of static feature for two frames image, that would disturb static feature point recognition. Because the translational motion optical flow orientations method judges with optical flow of single feature point and translational motion orientations, it is not reliable for static feature point recognition. Therefore, a 3D motion segmentation combined for static object imaging features recognition is proposed, which clusters the same optical flow of feature points in 3D motion subspace, including orientations and amplitude of translation and rotation. The simulation results show that the proposed static object imaging features recognition method improves the recognition correct rate and reliability of static features. So, it is effective for static object imaging features recognition in dynamic scene mapping.

Selective processing of all rotational and translational optic flow directions in the zebrafish pretectum and tectum

BackgroundThe processing of optic flow in the pretectum/accessory optic system allows animals to stabilize retinal images by executing compensatory optokinetic and optomotor behavior. The success of this behavior depends on the integration of information from both eyes to unequivocally identify all possible translational or rotational directions of motion. However, it is still unknown whether the precise direction of ego-motion is already identified in the zebrafish pretectum or later in downstream premotor areas.ResultsHere, we show that the zebrafish pretectum and tectum each contain four populations of motion-sensitive direction-selective (DS) neurons, with each population encoding a different preferred direction upon monocular stimulation. In contrast, binocular stimulation revealed the existence of pretectal and tectal neurons that are specifically tuned to only one of the many possible combinations of monocular motion, suggesting that further downstream sensory processing might not be needed to instruct appropriate optokinetic and optomotor behavior.ConclusionOur results suggest that local, task-specific pretectal circuits process DS retinal inputs and carry out the binocular sensory computations necessary for optokinetic and optomotor behavior.

Perceiving blurry scenes with translational optic flow, rotational optic flow or combined optic flow

Perceiving the spatial layout of objects is crucial in visual scene perception. Optic flow provides information about spatial layout. This information is not affected by image blur because motion detection uses low spatial frequencies in image structure. Therefore, perceiving scenes with blurry vision should be effective when optic flow is available. Furthermore, when blurry images and optic flow interact, optic flow specifies spatial relations and calibrates blurry images. Calibrated image structure then preserves spatial relations specified by optic flow after motion stops. Thus, perceiving blurry scenes should be stable when optic flow and blurry images are available. We investigated the types of optic flow that facilitate recognition of blurry scenes and evaluated the stability of performance. Participants identified scenes in blurry videos when viewing single frames and the entire videos that contained translational flow (Experiment 1), rotational flow (Experiment 2) or both (Experiment 3). When first viewing the blurry images, participants identified a few scenes. When viewing blurry video clips, their performance improved with translational flow, whether it was available alone or in combination with rotational flow. Participants were still able to perceive scenes from static blurry images one week later. Therefore, translational flow interacts with blurry image structures to yield effective and stable scene perception. These results imply that observers with blurry vision may be able to identify their surrounds when they locomote.

The role of lateral optic flow cues in hawkmoth flight control.

Flying animals require sensory feedback on changes of their body position, as well as on their distance from nearby objects. The apparent image motion, or optic flow, which is generated as animals move through the air, can provide this information. Flight tunnel experiments have been crucial for our understanding of how insects use optic flow for flight control in confined spaces. However, previous work mainly focused on species from two insect orders: Hymenoptera and Diptera. We therefore set out to investigate whether the previously described control strategies to navigate enclosed environments are also used by insects with a different optical system, flight kinematics and phylogenetic background. We tested the role of lateral visual cues for forward flight control in the hummingbird hawkmoth Macroglossum stellatarum (Sphingidae, Lepidoptera), which possesses superposition compound eyes, and has the ability to hover in addition to its capacity for fast forward flight. Our results show that hawkmoths use a similar strategy for lateral position control to bees and flies in balancing the magnitude of translational optic flow perceived in both eyes. However, the influence of lateral optic flow on flight speed in hawkmoths differed from that in bees and flies. Moreover, hawkmoths showed individually attributable differences in position and speed control when the presented optic flow was unbalanced.

Multi‐frame super‐resolution algorithm using common vector approach

Super-resolution (SR) applications aim to use information within one or more low-resolution (LR) image(s) to obtain high-resolution (HR) image(s). LR images might well be the consecutive frames of a video sequence. When multiple LR images are used as input, motion estimation (ME) between portions of images is an important step of the solution for this ill-posed problem. In this study, the authors employed translational optical flow for ME, followed by common vector approach (CVA) for HR image reconstruction from multiple sources. CVA provides a way to reduce outliers caused by noise, occlusion, shadows and incorrect ME. HR image blocks are obtained by combining common and difference vectors of the blocks’ class that are handled separately. Noise in difference vectors is reduced by a known noise reduction method before combining. Separate handling of common and difference parts guarantees better results and greatly reduce artefacts. Compared to the state-of-the-art, experimental results confirm the authors’ achievement by visual, peak signal-to-noise ratio and structural similarity index measures criteria.

Identifying blurry scenes with translational optic flow, rotational optic flow or combined optic flow

Identifying blurry scenes with translational optic flow, rotational optic flow or combined optic flow

The contribution of viewpoint oscillations to the perception of distance travelled from optic flow

When exploring their environment, humans and other animals have the ability to use many sources of information to estimate the distance they travel. Several studies have shown that optic flow is a significant cue to perceive distance travelled. Furthermore, it was found that adding various viewpoint oscillations to a purely translational optic flow, simulating forward self-motion, modulated this perception. In a series of experiments, we tested whether the perception of distance travelled was also affected by viewpoint oscillation, similar to head motion during natural walking. A first series of experiments, participants were exposed to an immersive optic flow simulating forward self-motion and they were asked to indicate when they thought they had reached the remembered position of a previously seen target. The main conclusion from these experiments is that viewpoint oscillations improve the perception of distance travelled. Moreover, this effect appears to be linked to an increase in global retinal motion. However, the contribution cannot be ruled out. Manipulating different viewpoint oscillation frequencies showed that the optimal performance was observed for a range of frequencies close to that observed for the head motion during natural walking. Two further experiments aimed to test whether the idiosyncrasy of viewpoint oscillations affects the perception of distance travelled in stationary observers and whether the absence of their own viewpoint oscillation played an important role in subjects' estimates, while they were walking on a treadmill. And finally, in a last experiment we tried to develop a dynamic measure of distance travelled to a previously seen target, with a continuous pointing task method. Overall, our results show that viewpoint oscillations play an important role in visual self-motion perception and that several parameters (including visual information, proprioceptive information and ecological aspects of natural walking) seem to be involved in this process.

Spatial Encoding of Translational Optic Flow in Planar Scenes by Elementary Motion Detector Arrays

Elementary Motion Detectors (EMD) are well-established models of visual motion estimation in insects. The response of EMDs are tuned to specific temporal and spatial frequencies of the input stimuli, which matches the behavioural response of insects to wide-field image rotation, called the optomotor response. However, other behaviours, such as speed and position control, cannot be fully accounted for by EMDs because these behaviours are largely unaffected by image properties and appear to be controlled by the ratio between the flight speed and the distance to an object, defined here as relative nearness. We present a method that resolves this inconsistency by extracting an unambiguous estimate of relative nearness from the output of an EMD array. Our method is suitable for estimation of relative nearness in planar scenes such as when flying above the ground or beside large flat objects. We demonstrate closed loop control of the lateral position and forward velocity of a simulated agent flying in a corridor. This finding may explain how insects can measure relative nearness and control their flight despite the frequency tuning of EMDs. Our method also provides engineers with a relative nearness estimation technique that benefits from the low computational cost of EMDs.

Path Integration: Combining Optic Flow with Compass Orientation

The discovery of translational optic flow detectors in the central complex of a bee has inspired a new model of path integration.

Viewpoint oscillation frequency influences the perception of distance travelled from optic flow

In everyday life, humans and most animals need to navigate in their environment, which produces multiple sources of perceptual information, such as locomotor cues (i.e. proprioceptive, efference copy and vestibular cues) and optic flow. However, few studies focused on the role of the visual consequences of walking (bob, sway, and lunge head motion) on self-motion perception. In a previous study, in which static observers were confronted to a visual simulation of forward motion, we have shown that adding rhythmical components to an optic flow pattern improved the accuracy of subjects' travelled distance estimations, in comparison with a purely translational flow. These results were attributable to the fact that oscillations may increase the global retinal motion and thus improve vection. Another hypothesis was that, walking step frequency being a significant cue in speed perception, visual consequences of step frequency might be at the origin of better estimations.To test this, we used the same experimental procedure in which observers, immersed inside a 4-sided CAVE, had to indicate when they thought they had reached a previously seen target. We tested whether different oscillation frequencies would affect the perception of distance travelled. Observers were confronted with 4 conditions of optic flows simulating forward self-motion. The first condition was generated by purely translational optic flow, at constant speed. The three other conditions of flows were vertical triangular oscillations with three kinds of frequencies added to linear forward motion, at the same forward speed. Results show that two groups can be distinguished. Regarding the first group, as in the previous study, adding rhythmic components improves the perception of distance travelled. For the second group, the higher the frequency, the earlier the answers, suggesting that these subjects related the oscillation frequency to their step frequency and perceived themselves as moving faster.

How bumblebees use lateral and ventral optic flow cues for position control in environments of different proximity

Flying insects frequently navigate through environments of different complexity. In this study, buff-tailed bumblebees (Bombus terrestris L.) were trained to fly along tunnels of different widths, from 60 to 240 cm. In tunnel widths of 60 and 120 cm, bumblebees control their lateral position by balancing the magnitude of translational optic flow experienced in the lateral visual field of each eye. In wider tunnels, bumblebees use translational optic flow cues in the ventral visual field to control their lateral position and to steer along straight tracks. Our results also suggest that bumblebees prefer to fly over surfaces that provide strong ventral optic flow cues, rather than over featureless ones. Together, these strategies allow bumblebees to minimize the risk of collision and to maintain relatively straight flight paths in a broad range of environments.