Optical Flow Estimation Method Research Articles

Robust optical flow estimation involving exponential fractional-order derivatives

This work presents a fractional-order method for optical flow estimation. The proposed method is based on the Classic+NL algorithm and the Caputo-Fabrizio fractional-order derivative. Therefore, the proposed operator can be seen as a generalization of this integer-order model. The performance of the algorithm is analyzed by using the Average End Point Error (EPE) and the Average Angular Error (AAE) considering the MPI Sintel and Middlebury Datasets. Experimental results on these dataset show that the proposed algorithm performs better than other existing techniques through the analysis of the two error metrics mentioned above. The firefly optimization algorithm was applied to obtain the fractional-order, which minimizes the EPE, for each of images sequence used.

Graph Laplacian Regularization for Robust Optical Flow Estimation.

This paper proposes graph Laplacian regularization for robust estimation of optical flow. First, we analyze the spectral properties of dense graph Laplacians and show that dense graphs achieve a better trade-off between preserving flow discontinuities and filtering noise, compared with the usual Laplacian. Using this analysis, we then propose a robust optical flow estimation method based on Gaussian graph Laplacians. We revisit the framework of iteratively reweighted least-squares from the perspective of graph edge reweighting, and employ the Welsch loss function to preserve flow discontinuities and handle occlusions. Our experiments using the Middlebury and MPI-Sintel optical flow datasets demonstrate the robustness and the efficiency of our proposed approach.

Robust optical flow estimation based on wavelet

Concentrating on the issue that the systematic error caused by variation of illumination conditions and sensor noise leads to poor robustness and low accuracy of optical flow calculation, based on the wavelet multi-resolution theory, a robust optical flow estimation method is developed in this paper. With the multi-resolution characteristics of wavelet, the systematic error is incorporated into the calculation of optical flow to improve the robustness and estimation accuracy. In what follows, the total least squares method can be exploited to solve the obtained overdetermined wavelet optical flow equations such that the optical flow vector can be achieved. As compared to the traditional Lucas–Kanade approach, Horn–Schunck method, and the optical flow estimation algorithm in omnidirectional images using wavelet approach, simulation results show that the proposed algorithm can significantly improve the accuracy of optical flow estimation and the robustness of the optical flow field.

3D Multi-Resolution Optical Flow Analysis of Cardiovascular Pulse Propagation in Human Brain.

The brain is cleaned from waste by glymphatic clearance serving a similar purpose as the lymphatic system in the rest of the body. Impairment of the glymphatic brain clearance precedes protein accumulation and reduced cognitive function in Alzheimer's disease (AD). Cardiovascular pulsations are a primary driving force of the glymphatic brain clearance. We developed a method to quantify cardiovascular pulse propagation in the human brain with magnetic resonance encephalography (MREG). We extended a standard optical flow estimation method to three spatial dimensions, with a multi-resolution processing scheme. We added application-specific criteria for discarding inaccurate results. With the proposed method, it is now possible to estimate the propagation of cardiovascular pulse wavefronts from the whole brain MREG data sampled at 10 Hz. The results show that on average the cardiovascular pulse propagates from major arteries via cerebral spinal fluid spaces into all tissue compartments in the brain. We present an example, that cardiovascular pulsations are significantly altered in AD: coefficient of variation and sample entropy of the pulse propagation speed in the lateral ventricles change in AD. These changes are in line with the theory of glymphatic clearance impairment in AD. The proposed non-invasive method can assess a performance indicator related to the glymphatic clearance in the human brain.

KalmanFlow 2.0: Efficient Video Optical Flow Estimation via Context-Aware Kalman Filtering.

Recent studies on optical flow typically focus on the estimation of the single flow field in between a pair of images but pay little attention to the multiple consecutive flow fields in a longer video sequence. In this paper, we propose an efficient video optical flow estimation method by exploiting the temporal coherence and context dynamics under a Kalman filtering system. In this system, pixel's motion flow is first formulated as a second-order time-variant state vector and then optimally estimated according to the measurement and system noise levels within the system by maximum a posteriori criteria. Specifically, we evaluate the measurement noise according to the flow's temporal derivative, spatial gradient, and warping error. We determine the system noise based on the similarity of contextual information, which is represented by the compact features learned by pre-trained convolutional neural networks. The context-aware Kalman filtering helps improve the robustness of our method against abrupt change of light and occlusion/dis-occlusion in complicated scenes. The experimental results and analyses on the MPI Sintel, Monkaa, and Driving video datasets demonstrate that the proposed method performs favorably against the state-of-the-art approaches.

A multi-aperture optical flow estimation method for an artificial compound eye

A multi-aperture optical flow estimation method for an artificial compound eye

High-resolution velocimetry from tracer particle fields using a wavelet-based optical flow method

A wavelet-based optical flow method for high-resolution velocimetry based on tracer particle images is presented. The current optical flow estimation method (WOF-X) is designed for improvements in processing experimental images by implementing wavelet transforms with the lifting method and symmetric boundary conditions. This approach leads to speed and accuracy improvements over the existing wavelet-based methods. The current method also exploits the properties of fluid flows and uses the known behavior of turbulent energy spectra to semi-automatically tune a regularization parameter that has been primarily determined empirically in the previous optical flow algorithms. As an initial step in evaluating the WOF-X method, synthetic particle images from a 2D DNS of isotropic turbulence are processed and the results are compared to a typical correlation-based PIV algorithm and previous optical flow methods. The WOF-X method produces a dense velocity estimation, resulting in an order-of-magnitude increase in velocity vector resolution compared to the traditional correlation-based PIV processing. Results also show an improvement in velocity estimation by more than a factor of two. The increases in resolution and accuracy of the velocity field lead to significant improvements in the calculation of velocity gradient-dependent properties such as vorticity. In addition to the DNS results, the WOF-X method is evaluated in a series of two-dimensional vortex flow simulations to determine optimal experimental design parameters. Recommendations for optimal conditions for tracer particle seed density and inter-frame particle displacement are presented. The WOF-X method produces minimal error at larger particle displacements and lower relative error over a larger velocity dynamic range as compared to correlation-based processing.

Ghost-removal image warping for optical flow estimation

Traditional image warping methods used in optical flow estimation usually adopt simple interpolation strategies to obtain the warped images. But without considering the characteristic of occluded regions, the traditional methods may result in undesirable ghosting artifacts. To tackle this problem, in this paper we propose a novel image warping method to effectively remove ghosting artifacts. To be Specific, when given a warped image, the ghost regions are firstly discriminated using the optical flow information. Then, we use a new image compensation technique to eliminate the ghosting artifacts. The proposed method can avoid serious distortion in the warped images, therefore can prevent error propagation in the coarse-to-fine optical flow estimation schemes. Meanwhile, our approach can be easily integrated into various optical flow estimation methods. Experimental results on some popular datasets such as Flying Chairs and MPI-Sintel demonstrate that the proposed method can improve the performance of current optical flow estimation methods.

Detection and analysis of spatiotemporal patterns in brain activity.

There is growing evidence that population-level brain activity is often organized into propagating waves that are structured in both space and time. Such spatiotemporal patterns have been linked to brain function and observed across multiple recording methodologies and scales. The ability to detect and analyze these patterns is thus essential for understanding the working mechanisms of neural circuits. Here we present a mathematical and computational framework for the identification and analysis of multiple classes of wave patterns in neural population-level recordings. By drawing a conceptual link between spatiotemporal patterns found in the brain and coherent structures such as vortices found in turbulent flows, we introduce velocity vector fields to characterize neural population activity. These vector fields are calculated for both phase and amplitude of oscillatory neural signals by adapting optical flow estimation methods from the field of computer vision. Based on these velocity vector fields, we then introduce order parameters and critical point analysis to detect and characterize a diverse range of propagating wave patterns, including planar waves, sources, sinks, spiral waves, and saddle patterns. We also introduce a novel vector field decomposition method that extracts the dominant spatiotemporal structures in a recording. This enables neural data to be represented by the activity of a small number of independent spatiotemporal modes, providing an alternative to existing dimensionality reduction techniques which separate space and time components. We demonstrate the capabilities of the framework and toolbox with simulated data, local field potentials from marmoset visual cortex and optical voltage recordings from whole mouse cortex, and we show that pattern dynamics are non-random and are modulated by the presence of visual stimuli. These methods are implemented in a MATLAB toolbox, which is freely available under an open-source licensing agreement.

A CNNs-based method for optical flow estimation with prior constraints and stacked U-Nets

Traditional approaches for optical flow estimation always build an energy function which contains data term and smoothness term. However, optimizing the complex function is usually time-consuming. Nowadays, convolution neural networks have been applied in optical flow area. Most of them use large dataset for learning optical flow end-to-end, which can learn motion information from a large amount of prior information prepared in advance. However, these methods rely excessively on the learning ability of the network while ignoring some of well-proven assumptions in traditional approaches. In this paper, inspired by traditional methods, we present a network for learning optical flow, which combines traditional constraints with a supervised network. In the process of network optimization, the brightness constancy, gradient constancy and spatial smoothness assumptions are used to guide the training of network. Moreover, we stack several sub-networks integrated with prior constraints to form a large network for iterative refinement. Our method is tested on several public datasets, such as MPI-Sintel, KITTI2012, KITTI2015, Middlebury. The experimental results show that adding the prior constraints during training can obtain more refined and accurate flow. Compared with other recent methods, our method can achieve state-of-the-art performance on several public benchmarks.

Spiking Optical Flow for Event-Based Sensors Using IBM's TrueNorth Neurosynaptic System.

This paper describes a fully spike-based neural network for optical flow estimation from dynamic vision sensor data. A low power embedded implementation of the method, which combines the asynchronous time-based image sensor with IBM's TrueNorth Neurosynaptic System, is presented. The sensor generates spikes with submillisecond resolution in response to scene illumination changes. These spike are processed by a spiking neural network running on TrueNorth with a 1-ms resolution to accurately determine the order and time difference of spikes from neighbouring pixels, and therefore infer the velocity. The spiking neural network is a variant of the Barlow Levick method for optical flow estimation. The system is evaluated on two recordings for which ground truth motion is available, and achieves an average endpoint error of 11% at an estimated power budget of under 80mW for the sensor and computation.

A Novel Approach for the Short-Term Forecast of the Effective Cloud Albedo

The increasing use of renewable energies as a source of electricity has led to a fundamental transition of the power supply system. The integration of fluctuating weather-dependent energy sources into the grid already has a major impact on its load flows. As a result, the interest in forecasting wind and solar radiation with a sufficient accuracy over short time periods (<4 h) has grown. In this study, the short-term forecast of the effective cloud albedo based on optical flow estimation methods is investigated. The optical flow method utilized here is TV-L1 from the open source library OpenCV. This method uses a multi-scale approach to capture cloud motions on various spatial scales. After the clouds are displaced, the solar surface radiation will be calculated with SPECMAGIC NOW, which computes the global irradiation spectrally resolved from satellite imagery. Due to the high temporal and spatial resolution of satellite measurements, the effective cloud albedo and thus solar radiation can be forecasted from 5 min up to 4 h with a resolution of 0.05°. The validation results of this method are very promising, and the RMSE of the 30-min, 60-min, 90-min and 120-min forecast equals 10.47%, 14.28%, 16.87% and 18.83%, respectively. The paper gives a brief description of the method for the short-term forecast of the effective cloud albedo. Subsequently, evaluation results will be presented and discussed.

Comparative Analysis of Tracking Objects Using Optical Flow and Background Estimation on Silent Camera

Tracking and object is one of the utilizations on the field of the computer vision application. Object tracking utilization as a computer vision in this study is used to identify objects which exist within a frame and calculate the number of objects passing within a frame. The utilization of computer vision in various fields of application can be used to solve the existing problems. The method used in object tracking is by comparison between optical flow estimation method with background method. The test is conducted by using a still camera for both methods by making changes to the parameter values used as a reference. The results of the tests, conducted on the three video objects by comparing the two methods show a Total Recorded Time better than those of the background estimation method, being smaller than 100 seconds. Testing both methods successfully identifies the object tracking and calculates the number of passing cars.

A neural-based method for optical flow estimation using phase correlation

Motion estimation for image sequences is one of the most important tasks in computer vision. Thus, many methods have been proposed to solve this problem, but even so, it still lacks a generic method that determines motion in all situations and for all types of objects. In this work, we propose a two-phase connectionist neural method for motion estimation in the frequency domain that takes discontinuities into account. In the first phase, the most probable motion of each pixel is estimated using self-organising maps principles and the phase correlation method. The second phase consists in regularising the displacement field that considers the discontinuities. When tested and compared with other approaches on both synthetic and real image sequences, our method showed good performances according to the following criteria: precision, regularity, resistance to noise and running-time. Moreover, it could estimate the motions in cases where rotation and scaling are required.

A neural-based method for optical flow estimation using phase correlation

Motion estimation for image sequences is one of the most important tasks in computer vision. Thus, many methods have been proposed to solve this problem, but even so, it still lacks a generic method that determines motion in all situations and for all types of objects. In this work, we propose a two-phase connectionist neural method for motion estimation in the frequency domain that takes discontinuities into account. In the first phase, the most probable motion of each pixel is estimated using self-organising maps principles and the phase correlation method. The second phase consists in regularising the displacement field that considers the discontinuities. When tested and compared with other approaches on both synthetic and real image sequences, our method showed good performances according to the following criteria: precision, regularity, resistance to noise and running-time. Moreover, it could estimate the motions in cases where rotation and scaling are required.

A new method for the optical flow estimation and segmentation of moving objects 'NMES'

Segmentation of moving objects in video sequence is essential task in computer vision. This paper focuses on developing a new method for discriminate moving objects from a static background, focusing on the combination of motion, colour and texture features. First, we have used block-matching for computing the optical flow, we also have taken in consideration the result of frame difference, to improve the quality of the optical flow. Moreover, we have used the k-means clustering algorithm owing to group the pixels, having similar features. Second, the result of the grouping pixels is used as an input in Chan-Vese model, in order to attract the evolving contour of moving objects contours. To evaluate the performance of our proposed method, we experiment it on challenging sequences. It has shown that our method provides an improved segmentation results.

Vertical landing for micro air vehicles using event‐based optical flow

AbstractSmall flying robots can perform landing maneuvers using bio‐inspired optical flow by maintaining a constant divergence. However, optical flow is typically estimated from frame sequences recorded by standard miniature cameras. This requires processing full images onboard, which limits the update rate of divergence measurements and thus the speed of the control loop and the robot. Event‐based cameras overcome these limitations by only measuring pixel‐level brightness changes at microsecond temporal accuracy, hence providing an efficient mechanism for optical flow estimation. This paper presents, to the best of our knowledge, the first work integrating event‐based optical flow estimation into the control loop of a flying robot. We extend an existing “local plane fitting” algorithm to obtain an improved and more computationally efficient optical flow estimation method, which is valid for a wide range of optical flow velocities. This method is validated for real event sequences. In addition, a method for estimating the divergence from event‐based optical flow is introduced that accounts for the aperture problem. The developed algorithms are implemented in a constant divergence landing controller onboard a quadrotor. Experiments show that, using event‐based optical flow, accurate divergence estimates can be obtained over a wide range of speeds. This enables the quadrotor to perform very fast landing maneuvers.

A novel method for tracking and analysis of EEG activation across brain lobes

Abstract Electroencephalography (EEG) measures the electrical activity of brain that is generated by the synchronized activity of thousands of neurons. In this paper, our first goal is to develop a novel method to track the EEG activation in different brain regions involved in the processing of target and non-target stimuli in oddball paradigm. Secondly we want to identify the difference in the pattern of activation for different oddball tasks. The EEG data has been acquired from twenty healthy volunteers for the visual oddball experiment. In the task two types of visual stimuli target (rare) and non-target (frequent) were randomly presented. The subjects were instructed to press the enter button when they identify the target stimuli. The EEG data acquired is converted into EEG topo-maps. In our method the flow of activation between consecutive topo-maps is estimated by using Horn and Schunck Optical Flow estimation method. It helps to generate the motion field between consecutive topo-maps which is considered as flow of activation between two time frames. Different motion vectors are clustered into a group based on the activation level. These clusters are tracked between different frames as a measure of the activation flow. Finally we analyze the flow of activation across different brain lobes for different cases encountered in the Oddball paradigm by plotting average activation graph with respect to time. Analysis of the data has revealed that high activation has been observed in the Frontal and Occipital lobes in general for the oddball task. Frontal lobe shows high activation for target with response case, this is followed by the Occipital lobe. The activation in frontal lobe starts increasing from frame no 60 (240 ms) (out of 125 frames). Occipital lobe shows high activation for target with no response and no target no response cases in the region from 40 to 60 frames. Parietal lobe shows high activation for target with response near the end of task from frame no. 100 onwards. Hence, we have been able to identify different patterns in the activation flow that differentiates different Oddball tasks. This activation pattern is consistent with the event related potential signal generated by the Oddball paradigm. The pattern for the individual subjects also follow the average pattern of high activation in the frontal and occipital region for target and non target stimuli. We have also used cross correlation (a classical connectivity method) for comparison of the results. A subjective comparison of the results show that our proposed method is capable of tracking the EEG activation.

Variational method for joint optical flow estimation and edge-aware image restoration

The most popular optical flow algorithms rely on optimizing the energy function that integrates a data term and a smoothness term. In contrast to this traditional framework, we derive a new objective function that couples optical flow estimation and image restoration. Our method is inspired by the recent successes of edge-aware constraints (EAC) in preserving edges in general gradient domain image filtering. By incorporating an EAC image fidelity term (IFT) in the conventional variational model, the new energy function can simultaneously estimate optical flow and restore images with preserved edges, in a bidirectional manner. For the energy minimization, we rewrite the EAC into gradient form and optimize the IFT with Euler–Lagrange equations. We can thus apply the image restoration by analytically solving a system of linear equations. Our EAC-combined IFT is easy to implement and can be seamlessly integrated into various optical flow functions suggested in literature. Extensive experiments on public optical flow benchmarks demonstrate that our method outperforms the current state-of-the-art in optical flow estimation and image restoration.

Monocular, Boundary-Preserving Joint Recovery of Scene Flow and Depth

Variational joint recovery of scene flow and depth from a single image sequence, rather than from a stereo sequence as others required, was investigated in Mitiche et al. (2015) using an integral functional with a term of conformity of scene flow and depth to the image sequence spatiotemporal variations, and L2 regularization terms for smooth depth field and scene flow. The resulting scheme was analogous to the Horn and Schunck optical flow estimation method except that the unknowns were depth and scene flow rather than optical flow. Several examples were given to show the basic potency of the method: It was able to recover good depth and motion, except at their boundaries because L2 regularization is blind to discontinuities which it smooths indiscriminately. The method we study in this paper generalizes to L1 regularization the formulation of Mitiche et al. (2015) so that it computes boundary preserving estimates of both depth and scene flow. The image derivatives, which appear as data in the functional, are computed from the recorded image sequence also by a variational method which uses L1 regularization to preserve their discontinuities. Although L1 regularization yields nonlinear Euler-Lagrange equations for the minimization of the objective functional, these can be solved efficiently. The advantages of the generalization, namely sharper computed depth and three-dimensional motion, are put in evidence in experimentation with real and synthetic images which shows the results of L1 versus L2 regularization of depth and motion, as well as the results using L1 rather than L2 regularization of image derivatives.