Optic Flow Processing Research Articles

Tracking cortical entrainment to stages of optic-flow processing

In human visual processing, information from the visual field passes through numerous transformations before perceptual attributes such as motion are derived. Determining the sequence of transforms involved in the perception of visual motion has been an active field since the 1940s. One plausible family of models are the spatiotemporal energy models, based on computations of motion energy computed from the spatiotemporal features the visual field. One of the most venerated is that of Heeger (1988), which hypotheses that motion is estimated by matching the predicted spatiotemporal energy in frequency space. In this study, we investigate the plausibility of Heeger’s model by testing for evidence of cortical entrainment to its components. Entrainment of cortical activity to these components was estimated using measurements of electro- and magnetoencephalographic (EMEG) activity, recorded while healthy subjects watched videos of dots moving left and right across their visual field. We find entrainment to several components of Heeger’s model bilaterally in occipital lobe regions, including representations of motion energy at a latency of 80 ms, overall velocity at 95 ms, and acceleration at 130 ms. We find little evidence of entrainment to displacement. We contrast Heeger’s biologically inspired model with alternative baseline models, finding that Heeger’s model provides a closer fit to the observed data. These results help shed light on the processes through which perception of motion arises in the visual processing stream.

Encoding of global visual motion in the avian pretectum shifts from a bias for temporal-to-nasal selectivity to omnidirectional excitation across speeds.

The pretectum of vertebrates contains neurons responsive to global visual motion. These signals are sent to the cerebellum, forming a subcortical pathway for processing optic flow. Global motion neurons exhibit selectivity for both direction and speed, but this is usually assessed by first determining direction preference at intermediate velocity (16-32 deg/sec), and then assessing speed tuning at the preferred direction. A consequence of this approach is that it is unknown if and how direction preference changes with speed. We measured directional selectivity in 114 pretectal neurons from 44 zebra finches (Taeniopygia guttata) across spatial and temporal frequencies, corresponding to a speed range of 0.062 to 1024°/s. Pretectal neurons were most responsive at 32-64°/s with lower activity as speed increased or decreased. At each speed, we determined if cells were directionally-selective, bidirectionally-selective, omnidirectionally responsive, or unmodulated. Notably, at 32°/s, 60% of the cells were directionally selective and 28% were omnidirectionally responsive. In contrast, at 1024°/s, 20% of the cells were directionally selective and nearly half of the population was omnidirectionally responsive. Only 15% of the cells were omnidirectionally excited across most speeds. The remaining 85% of the cells had direction tuning that changed with speed. Collectively, these results indicate a shift from a bias for directional tuning at intermediate speeds of global visual motion to a bias for omnidirectional responses at faster speeds. These results suggest a potential role for the pretectum during flight by detecting unexpected drift or potentials collisions, depending on the speed of the optic flow signal.Significance Statement During locomotion, images of edges and surfaces in the environment move across the retina, a signal of global visual motion called optic flow. Retinal recipient areas in the accessory optic system and the pretectum are the earliest sites to encode this signal, and the neurons are selective for direction and speed. Previous work suggested that directional selectivity may change across speeds but this has never been systematically studied. We measured direction preferences from 0.062 to 1024°/s in the avian pretectum. We found that pretectal global motion neurons are biased for temporal-to-nasal motion at intermediate speeds but biased for omnidirectional responses at faster speeds. These results suggest the pretectum could function to detect both unexpected drift and potential collisions during locomotion.

Bilateral interactions of optic-flow sensitive neurons coordinate course control in flies

Animals rely on compensatory actions to maintain stability and navigate their environment efficiently. These actions depend on global visual motion cues known as optic-flow. While the optomotor response has been the traditional focus for studying optic-flow compensation in insects, its simplicity has been insufficient to determine the role of the intricate optic-flow processing network involved in visual course control. Here, we reveal a series of course control behaviours in Drosophila and link them to specific neural circuits. We show that bilateral electrical coupling of optic-flow-sensitive neurons in the fly’s lobula plate are required for a proper course control. This electrical interaction works alongside chemical synapses within the HS-H2 network to control the dynamics and direction of turning behaviours. Our findings reveal how insects use bilateral motion cues for navigation, assigning a new functional significance to the HS-H2 network and suggesting a previously unknown role for gap junctions in non-linear operations.

Multi-YOLOv8: An infrared moving small object detection model based on YOLOv8 for air vehicle

The detection of infrared moving small objects faces significant challenges in the field of object detection for air vehicles. These types of objects usually occupy a small number of pixels in an infrared image, resulting in limited feature information, considerable feature loss, low recognition accuracy, and various challenges in single-frame detection. To address these challenges, this paper proposes an efficient multi-input method named Multi-YOLOv8, which is based on the YOLOv8s model. The proposed method uses current frames as a primary input and incorporates optical flow processing images and background suppression images as auxiliary inputs to improve detection performance. In addition, an improved method is developed for optical flow computations, named the pyramidal weight-momentum Horn–Schunck (PWMHS) method, which can process optical flows efficiently and precisely. An improved version of the Wise-IoU (WIoU) v3, referred to as α*-WIoU v3, is proposed as a bounding box regression (BBR) loss function to optimize the YOLOv8 network. Further, the BiFormer module and lightweight convolution GSConv are introduced to improve the attention to key information for the objects and balance the computational cost and detection performance, respectively. Moreover, a small object detection layer is added the YOLOv8 network to improve the capability for small object detection. Finally, a warming-up training method that can reduce the dependency on auxiliary inputs and ensure model stability in case of auxiliary input failures is developed. The results of the comprehensive experiments on an open-access dataset reveal that the proposed model outperforms the mainstream models in overall performance. The proposed method can significantly enhance the detection ability of infrared moving small objects.

Processing of complex traffic scenes for effective steering and collision avoidance: a perspective, from research into human control, on the challenges for sensor-based autonomous vehicles on urban roads.

An overview is provided of behavioral research into human steering and collision avoidance including the processing of optic flow, optical looming and the role of the human mobile gaze system. A consideration is then made of the issues that may occur for autonomous vehicles (AV) when they move from grid-type road networks into complex inner-city streets and interact with human drivers, pedestrians and cyclists. Comparisons between human processing and AV processing of these interactions are made. This raises issues as to whether AV control systems need to mimic human visual processing more closely and highlights the need for AV systems to develop a "theory of road users" that allows attribution of intent to other drivers, cyclists or pedestrians. Guidelines for the development of a "theory of road users" for AVs are suggested.

Processing of translational, radial and rotational optic flow in older adults

Aging impacts human observer’s performance in a wide range of visual tasks and notably in motion discrimination. Despite numerous studies, we still poorly understand how optic flow processing is impacted in healthy older adults. Here, we estimated motion coherence thresholds in two groups of younger (age: 18–30, n = 42) and older (70–90, n = 42) adult participants for the three components of optic flow (translational, radial and rotational patterns). Stimuli were dynamic random-dot kinematograms (RDKs) projected on a large screen. Participants had to report their perceived direction of motion (leftward versus rightward for translational, inward versus outward for radial and clockwise versus anti-clockwise for rotational patterns). Stimuli had an average speed of 7°/s (additional recordings were performed at 14°/s) and were either presented full-field or in peripheral vision. Statistical analyses showed that thresholds in older adults were similar to those measured in younger participants for translational patterns, thresholds for radial patterns were significantly increased in our slowest condition and thresholds for rotational patterns were significantly decreased. Altogether, these findings support the idea that aging does not lead to a general decline in visual perception but rather has specific effects on the processing of each optic flow component.

Recasting visual areas specialized for processing optic flow in the human brain

Recasting visual areas specialized for processing optic flow in the human brain

Representation of Motion Direction in Visual Area MT Accounts for High Sensitivity to Centripetal Motion, Aligning with Efficient Coding of Retinal Motion Statistics.

The overrepresentation of centrifugal motion in the middle temporal visual area (area MT) has long been thought to provide an efficient coding strategy for optic flow processing. However, this overrepresentation compromises the detection of approaching objects, which is essential for survival. In the present study, we revisited this long-held notion by reanalyzing motion selectivity in area MT of three macaque monkeys (two males, one female) using random-dot stimuli instead of spot stimuli. We found no differences in the number of neurons tuned to centrifugal versus centripetal motion; however, centrifugally tuned neurons showed stronger tuning than centripetally tuned neurons. This was attributed to the heightened suppression of responses in centrifugal neurons to centripetal motion compared with that of centripetal neurons to centrifugal motion. Our modeling implies that this intensified suppression accounts for superior detection performance for weak centripetal motion stimuli. Moreover, through Fisher information analysis, we establish that the population sensitivity to motion direction in peripheral vision corresponds well with retinal motion statistics during forward locomotion. While these results challenge established concepts, considering the interplay of logarithmic Gaussian receptive fields and spot stimuli can shed light on the previously documented overrepresentation of centrifugal motion. Significantly, our findings reconcile a previously found discrepancy between MT activity and human behavior, highlighting the proficiency of peripheral MT neurons in encoding motion direction efficiently.SIGNIFICANCE STATEMENT The efficient coding hypothesis states that sensory neurons are tuned to specific, frequently experienced stimuli. Whereas previous work has found that neurons in the middle temporal (MT) area favor centrifugal motion, which results from forward locomotion, we show here that there is no such bias. Moreover, we found that the response of centrifugal neurons for centripetal motion was more suppressed than that of centripetal neurons for centrifugal motion. Combined with modeling, this provides a solution to a previously known discrepancy between reported centrifugal bias in MT and better detection of centripetal motion by human observers. Additionally, we show that population sensitivity in peripheral MT neurons conforms to an efficient code of retinal motion statistics during forward locomotion.

Extracting high-precision full-field displacement from videos via pixel matching and optical flow

Accurate sensing of full-field displacements plays a significant role in dynamic testing for structural health monitoring (SHM). Recently, video camera has become a more promising tool in displacement sensing due to advantages such as low price, agility, simultaneous measurement, and high spatial sensing resolution. Traditional target tracking approaches (e.g., digital image correlation, template matching, etc.) typically require clear targets on the structure surface and thus only provide sparse displacement. Phase-based motion extraction enables full-field displacement measurement at subpixel precision without the need of markers as tracking targets, which has achieved great success in structural modal identification among many other SHM applications. However, the phase-based approach in a Eulerian framework fails to deal with large motions. To tackle these challenges, an enhanced phase-based method, that synergistically combines the developed local amplitude supplemented pixel matching algorithm and optical flow processing, is proposed for full-field displacement sensing with subpixel precision meanwhile capable of dealing with large motions. In particular, the pixel matching estimates the integer-pixel motion vectors of pixels which have sufficient texture contrast determined by local amplitude. The optical flow, represented by the first order Taylor series, is then employed to approximate the rest subpixel motion associated with the integer-pixel motion vectors. The proposed technique is validated by processing a field-test videos with LVDT reference and then applied to other several lab/field recorded videos of different vibrational objects for displacement extraction and modal identification.

S$^{2}$ Loop: A Lightweight Spectral-Spatio Loop Closure Detector for Resource-Constrained Platforms

Visual loop closure detection is an essential backend task for long-term vSLAM applications. However, prior works cannot simultaneously meet the requirements of high recall and low computing and memory overhead, which prohibits their applicability to resource-constrained platforms. In this work we propose S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop, a novel spectral-spatio loop detector that explores an efficient loop searching method based on spectral image analysis and direct image alignment. S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop digests an image by its Fourier spectrum which provides a compact image hash value for fast candidate selection. The spectra also enable a convenient method of image alignment based on Fourier phase correction. The aligned images are further matched by an optical-flow process at the pixel granularity, which precisely estimates the frame dissimilarity against the turbulence caused by the perspective warping of random 3D scenes. According to our benchmark experiments on various datasets, S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop shows two orders of magnitude of less computing and memory overhead with comparable precision and recall over the prior DBoW method on a desktop PC. It is also much faster than the prior Fern method that has low computing complexity, and exhibits a much higher recall. While querying loops in 1000 runtime keyframes, S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop deployed onto a low-end STM32F4 MCU achieves 64fps.

Topography of optic flow processing in olivo-cerebellar pathways in zebra finches (Taeniopygia guttata).

In birds, the nucleus of the basal optic root (nBOR) and the nucleus lentiformis mesencephali (LM) are brainstem nuclei involved in the analysis of optic flow. A major projection site of both nBOR and LM is the medial column of the inferior olive (IO), which provides climbing fibers to the vestibulocerebellum. This pathway has been well documented in pigeons, but not other birds. Recent works have highlighted that zebra finches show specializations with respect to optic flow processing, which may be reflected in the organization of optic flow pathways to the IO. In this study, we characterized the organization of these pathways in zebra finches. We found that the medial column consists of at least eight subnuclei (i-viii) visible in Nissl-stained tissue. Using anterograde traces we found that the projections from LM and nBOR to the IO were bilateral, but heavier to the ipsilateral side, and showed a complementary pattern: LM projected to subnucleus i, whereas nBOR projected to subnuclei ii and v. Using retrograde tracers, we found that these subnuclei (i, ii and v) projected to the vestibulocerebellum (folia IXcd and X), whereas the other subnuclei projected to IXab and the lateral margin of VII and VIII. The nBOR also projected ipsilaterally to the caudo-medial dorsal lamella of the IO, which the retrograde experiments showed as projecting to the medial margin of VII and VIII. We compare these results with previous studies in other avian species.

Walking humans and running mice: perception and neural encoding of optic flow during self-motion.

Locomotion produces full-field optic flow that often dominates the visual motion inputs to an observer. The perception of optic flow is in turn important for animals to guide their heading and interact with moving objects. Understanding how locomotion influences optic flow processing and perception is therefore essential to understand how animals successfully interact with their environment. Here, we review research investigating how perception and neural encoding of optic flow are altered during self-motion, focusing on locomotion. Self-motion has been found to influence estimation and sensitivity for optic flow speed and direction. Nonvisual self-motion signals also increase compensation for self-driven optic flow when parsing the visual motion of moving objects. The integration of visual and nonvisual self-motion signals largely follows principles of Bayesian inference and can improve the precision and accuracy of self-motion perception. The calibration of visual and nonvisual self-motion signals is dynamic, reflecting the changing visuomotor contingencies across different environmental contexts. Throughout this review, we consider experimental research using humans, non-human primates and mice. We highlight experimental challenges and opportunities afforded by each of these species and draw parallels between experimental findings. These findings reveal a profound influence of locomotion on optic flow processing and perception across species. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Novel Solution of Topological Recognition of Indoor Objects Based on Optical Flow and Planar Attributes

An approach of qualitative optical flow processing for indoor object recognition based on planar attributes is presented. The qualitative processing is performed under hierarchical segmentations of optical flow vectors. The proposed solution for indoor object recognition is undertaken from identifying planar and atilt properties of optical flow images. The advantages of the proposed solutions are the use of much simpler arithmetic to obtain more 3D details about indoor objects.

Optic Flow Processing in Patients With Macular Degeneration.

Optic flow processing was characterized in patients with macular degeneration (MD). Twelve patients with dense bilateral scotomas and 12 age- and gender-matched control participants performed psychophysical experiments. Stimuli were dynamic random-dot kinematograms projected on a large screen. For each component of optic flow (translational, radial, and rotational), we estimated motion coherence discrimination thresholds in our participants using an adaptive Bayesian procedure. Thresholds for translational, rotational, and radial patterns were comparable between patients and their matched control participants. A negative correlation was observed in patients between the time since MD diagnosis and coherence thresholds for translational patterns. Our results suggest that in patients with MD, selectivity to optic flow patterns is preserved.

Matched function of the neuropil processing optic flow in flies and crabs: the lobula plate mediates optomotor responses in Neohelice granulata.

When an animal rotates (whether it is an arthropod, a fish, a bird or a human) a drift of the visual panorama occurs over its retina, termed optic flow. The image is stabilized by compensatory behaviours (driven by the movement of the eyes, head or the whole body depending on the animal) collectively termed optomotor responses. The dipteran lobula plate has been consistently linked with optic flow processing and the control of optomotor responses. Crabs have a neuropil similarly located and interconnected in the optic lobes, therefore referred to as a lobula plate too. Here we show that the crabs' lobula plate is required for normal optomotor responses since the response was lost or severely impaired in animals whose lobula plate had been lesioned. The effect was behaviour-specific, since avoidance responses to approaching visual stimuli were not affected. Crabs require simpler optic flow processing than flies (because they move slower and in two-dimensional instead of three-dimensional space), consequently their lobula plates are relatively smaller. Nonetheless, they perform the same essential role in the visual control of behaviour. Our findings add a fundamental piece to the current debate on the evolutionary relationship between the lobula plates of insects and crustaceans.

Egomotion-related visual areas respond to goal-directed movements.

Integration of proprioceptive signals from the various effectors with visual feedback of self-motion from the retina is necessary for whole-body movement and locomotion. Here, we tested whether the human visual motion areas involved in processing optic flow signals simulating self-motion are also activated by goal-directed movements (as saccades or pointing) performed with different effectors (eye, hand, and foot), suggesting a role in visually guiding movements through the external environment. To achieve this aim, we used a combined approach of task-evoked activity and effective connectivity (PsychoPhysiological Interaction, PPI) by fMRI. We localized a set of six egomotion-responsive visual areas through the flow field stimulus and distinguished them into visual (pIPS/V3A, V6+ , IPSmot/VIP) and visuomotor (pCi, CSv, PIC) areas according to recent literature. We tested their response to a visuomotor task implying spatially directed delayed eye, hand, and foot movements. We observed a posterior-to-anterior gradient of preference for eye-to-foot movements, with posterior (visual) regions showing a preference for saccades, and anterior (visuomotor) regions showing a preference for foot pointing. No region showed a clear preference for hand pointing. Effective connectivity analysis showed that visual areas were more connected to each other with respect to the visuomotor areas, particularly during saccades. We suggest that visual and visuomotor egomotion regions can play different roles within a network that integrates sensory-motor signals with the aim of guiding movements in the external environment.

Estimating curvilinear self-motion from optic flow with a biologically inspired neural system* *This work was supported by The Office of Naval Research ONR N00014-18-1-2283.

Optic flow provides rich information about world-relative self-motion and is used by many animals to guide movement. For example, self-motion along linear, straight paths without eye movements, generates optic flow that radiates from a singularity that specifies the direction of travel (heading). Many neural models of optic flow processing contain heading detectors that are tuned to the position of the singularity, the design of which is informed by brain area MSTd of primate visual cortex that has been linked to heading perception. Such biologically inspired models could be useful for efficient self-motion estimation in robots, but existing systems are tailored to the limited scenario of linear self-motion and neglect sensitivity to self-motion along more natural curvilinear paths. The observer in this case experiences more complex motion patterns, the appearance of which depends on the radius of the curved path (path curvature) and the direction of gaze. Indeed, MSTd neurons have been shown to exhibit tuning to optic flow patterns other than radial expansion, a property that is rarely captured in neural models. We investigated in a computational model whether a population of MSTd-like sensors tuned to radial, spiral, ground, and other optic flow patterns could support the accurate estimation of parameters describing both linear and curvilinear self-motion. We used deep learning to decode self-motion parameters from the signals produced by the diverse population of MSTd-like units. We demonstrate that this system is capable of accurately estimating curvilinear path curvature, clockwise/counterclockwise sign, and gaze direction relative to the path tangent in both synthetic and naturalistic videos of simulated self-motion. Estimates remained stable over time while rapidly adapting to dynamic changes in the observer’s curvilinear self-motion. Our results show that coupled biologically inspired and artificial neural network systems hold promise as a solution for robust vision-based self-motion estimation in robots.

A Dynamic Efficient Sensory Encoding Approach to Adaptive Tuning in Neural Models of Optic Flow Processing.

This paper introduces a self-tuning mechanism for capturing rapid adaptation to changing visual stimuli by a population of neurons. Building upon the principles of efficient sensory encoding, we show how neural tuning curve parameters can be continually updated to optimally encode a time-varying distribution of recently detected stimulus values. We implemented this mechanism in a neural model that produces human-like estimates of self-motion direction (i.e., heading) based on optic flow. The parameters of speed-sensitive units were dynamically tuned in accordance with efficient sensory encoding such that the network remained sensitive as the distribution of optic flow speeds varied. In two simulation experiments, we found that model performance with dynamic tuning yielded more accurate, shorter latency heading estimates compared to the model with static tuning. We conclude that dynamic efficient sensory encoding offers a plausible approach for capturing adaptation to varying visual environments in biological visual systems and neural models alike.

Digitizing parking area availability information using Canny algorithm based on Android application

The Covid-19 pandemic is a factor in accelerating digital transformation in all aspects of community needs, especially in the parking area. Unmonitored and non-digitalized parking lots can cause delays that increase the transmission of Covid-19. In this study, a parking digitization system is created to make it easier for the public to know how dense is the area of ??the location to be visited, as well as a vacant area that can be occupied to park vehicles by users of the parking lot through a smartphone application. This study uses the Canny algorithm through an optical flow process captured by the IP Camera, next the data is processed and presented in the Android application. From the system that has been made, the system detects the presence of vehicles with an accuracy of above 95%.

Real-time optical flow processing on embedded GPU: an hardware-aware algorithm to implementation strategy

Determining the optical flow of a video is a compute-intensive task essential for computer vision. For achieving this processing in real time, the whole algorithm deployment chain must be thought of for efficiency first. The development is usually divided into two parts: first, designing an algorithm that meets precision constraints, then, implementing and optimizing its execution on the targeted platform. We argue that unifying those operations enhances performance on the embedded processor. This paper is based on an industrial use case of computer vision. The objective is to determine dense optical flow in real time on an embedded GPU platform: the Nvidia AGX Xavier. The CLG (combined local–global) optical flow method, initially chosen, is analyzed to understand the convergence speed of its underlying optimization problem. The Jacobi solver is selected for implementation because of its parallel nature. The whole multi-level processing is then ported to the GPU, using several specific optimization strategies. In particular, we analyze the impact of fusing the solver’s iterations with the roofline model. As a result, with a 30 W power budget, our implementation runs at 60FPS, on $$640 \times 512$$ images, with a four-level processing. Hopefully, this example should provide feedback on the issues that arise when trying to port a method to a parallel platform and serve for further implementations of computer vision algorithms on specialized hardware.