Optical Flow Computation Method Research Articles

Real-Time Optical Flow for Vehicular Perception With Low- and High-Resolution Event Cameras

Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. Thus, they allow for applications under high-speed motion and complex lighting conditions, where traditional frame-based sensors show their limits with blur and over- or under-exposed pixels. Thanks to these unique properties, they represent nowadays an highly attractive sensor for ITS-related applications. Event-based optical flow (EBOF) has been studied following the rise in popularity of these neuromorphic cameras. The recent arrival of high-definition neuromorphic sensors, however, challenges the existing approaches, because of the increased resolution of the events pixel array and a much higher throughput. As an answer to these points, we propose an optimized framework for computing optical flow in real-time with both low- and high-resolution event cameras. We formulate a novel dense representation for the sparse events flow, in the form of the “inverse exponential distance surface”. It serves as an interim frame, designed for the use of proven, state-of-the-art frame-based optical flow computation methods. We evaluate our approach on both low- and high-resolution driving sequences, and show that it often achieves better results than the current state of the art, while also reaching higher frame rates, 250Hz at 346 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times260$ </tex-math></inline-formula> pixels and 77Hz at 1280 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times720$ </tex-math></inline-formula> pixels.

Self-Attention-Based Multiscale Feature Learning Optical Flow With Occlusion Feature Map Prediction

Even though optical flow approaches based on convolutional neural networks have achieved remarkable performance with respect to both accuracy and efficiency, large displacements and motion occlusions remain challenges for most existing learning-based models. To address the abovementioned issues, we propose in this paper a self-attention-based multiscale feature learning optical flow computation method with occlusion feature map prediction. First, we exploit a self-attention mechanism-based multiscale feature learning module to compensate for large displacement optical flows, and the presented module is able to capture long-range dependencies from the input frames. Second, we design a simple but effective self-learning module to acquire an occlusion feature map, in which the predicted occlusion map is utilized to correct the optical flow estimation in occluded areas. Third, we explore a hybrid loss function that integrates the photometric and smoothness losses into the classical endpoint error (EPE)-based loss to ensure the accuracy and robustness of the presented network. Finally, we compare the proposed method with some state-of-the-art approaches using the MPI-Sintel and KITTI test databases. The experimental results demonstrate that the proposed method achieved competitive performance with respect to both accuracy and robustness, and it produced the better results compared to other methods under large displacements and motion occlusions.

Linking local and global optical flow computation by subspace regularization

Optical flow computation methods are traditionally classified to two categories: local and global. Several previous works have investigated the combination of them by exploiting their complementary effects. Unlike previous works, this paper is motivated by the common weakness of the two schemes and proposes linking local and global computation by subspace regularization. We will show that the subspace regularization can effectively reduce the error caused by ill-conditioned local computation systems, while inhibiting error diffusion suffered by the global regularization. Experimental results on benchmark sequences validate the enhanced performance of the proposed combination scheme.

Action Recognition by Joint Spatial-Temporal Motion Feature

This paper introduces a method for human action recognition based on optical flow motion features extraction. Automatic spatial and temporal alignments are combined together in order to encourage the temporal consistence on each action by an enhanced dynamic time warping (DTW) algorithm. At the same time, a fast method based on coarse-to-fine DTW constraint to improve computational performance without reducing accuracy is induced. The main contributions of this study include (1) a joint spatial-temporal multiresolution optical flow computation method which can keep encoding more informative motion information than recent proposed methods, (2) an enhanced DTW method to improve temporal consistence of motion in action recognition, and (3) coarse-to-fine DTW constraint on motion features pyramids to speed up recognition performance. Using this method, high recognition accuracy is achieved on different action databases like Weizmann database and KTH database.

De-Interlacing Using Nonlocal Costs and Markov-Chain-Based Estimation of Interpolation Methods

A new method of de-interlacing is proposed. De-interlacing is revisited as the problem of assigning a sequence of interpolation methods (interpolators) to a sequence of missing pixels of an interlaced frame (field). With this assumption, our de-interlacing algorithm (de-interlacer), undergoes transitions from one interpolation method to another, as it moves from one missing pixel position to the horizontally adjacent missing pixel position in a missing row of a field. We assume a discrete countable-state Markov-chain model on the sequence of interpolators (Markov-chain states) which are selected from a user-defined set of candidate interpolators. An estimation of the optimum sequence of interpolators with the aforementioned Markov-chain model requires the definition of an efficient cost function as well as a global optimization technique. Our algorithm introduces for the first time using a nonlocal cost (NLC) scheme. The proposed algorithm uses the NLC to not only measure the fitness of an interpolator at a missing pixel position, but also to derive an approximation for transition matrix (TM) of the Markov-chain of interpolators. The TM in our algorithm is a frame-variate matrix, i.e., the algorithm updates the TM for each frame automatically. The algorithm finally uses a Viterbi algorithm to find the global optimum sequence of interpolators given the cost function defined and neighboring original pixels in hand. Next, we introduce a new MAP-based formulation for the estimation of the sequence of interpolators this time not by estimating the best sequence of interpolators but by successive estimations of the best interpolator at each missing pixel using Forward-Backward algorithm. Simulation results prove that, while competitive with each other on different test sequences, the proposed methods (one using Viterbi and the other Forward-Backward algorithm) are superior to state-of-the-art de-interlacing algorithms proposed recently. Finally, we propose motion compensated versions of our algorithm based on optical flow computation methods and discuss how it can improve the proposed algorithm.

OPTICAL FLOW FOR GLACIER MOTION ESTIMATION

Abstract. Quantitative measurements of glacier flow over time are an important ingredient for glaciological research, for example to determine the mass balances and the evolution of glaciers. Measuring glacier flow in multi-temporal images involves the estimation of a dense set of corresponding points, which in turn define the flow vectors. Furthermore glaciers exhibit rather difficult radiometry, since their surface usually contains homogeneous areas as well as weak texture and contrast. To date glacier flow is usually observed by manually measuring a sparse set of correspondences, which is labor-intensive and often yields rather irregular point distributions, with the associated problems of interpolating over large areas. In the present work we propose to densely compute motion vectors at every pixel, by using recent robust methods for optic flow computation. Determining the optic flow, i.e. the dense deformation field between two images of a dynamic scene, has been a classic, long-standing research problem in computer vision and image processing. Sophisticated methods exist to optimally balance data fidelity with smoothness of the motion field. Depending on the strength of the local image gradients these methods yield a smooth trade-off between matching and interpolation, thereby avoiding the somewhat arbitrary decision which discrete anchor points to measure, while at the same time mitigating the problem of gross matching errors. We evaluate our method by comparing with manually measured point wise ground truth.

Rapid Optical Flow Computation Based on Multigrid

In order to make optical flow methods suitable for real-time applications, this paper presents a rapid optical flow computation method based on multigrid. Multigrid computation method is well known for being the fastest numerical method when solving elliptic boundary-value problems, so we integrate multigrid into Horn-Schunck algorithm in order to obtain rapid computation. Firstly, we analyze the convergence process of traditional methods for optical flow computation; secondly, we combine the multigrid method with Horn-Schunck method, and in this way optical flow can be calculated efficiently; finally, the experiments show that our rapid optical flow computation method is efficient for target detection and video tracking systems.

Variational optic flow on the Sony PlayStation 3

While modern variational methods for optic flow computation offer dense flow fields and highly accurate results, their computational complexity has prevented their use in many real-time applications. With cheap modern parallel hardware such as the Cell Processor of the Sony PlayStation 3, new possibilities arise. For a linear and a nonlinear variant of the popular combined local-global method, we present specific algorithms on this architecture that are tailored towards real-time performance. They are based on bidirectional full multigrid methods with a full approximation scheme in the nonlinear setting. Their parallel design on the Cell hardware uses a temporal instead of a spatial decomposition, and processes operations in a vector-based manner. Memory latencies are reduced by a locality-preserving cache management and optimised access patterns. For images of size 316 × 252 pixels, we obtain dense flow fields for up to 210 frames per second.

Variational optic flow on the Sony PlayStation 3 – accurate dense flow fields for real-time applications

While modern variational methods for optic flow computation offer dense flow fields and highly accurate results, their computational complexity has prevented their use in many real-time applications. With cheap modern parallel hardware such as the Sony PlayStation 3 new possibilities arise. For a linear and a nonlinear variant of the popular combined local-global (CLG) method, we present specific algorithms that are tailored towards real-time performance. They are based on bidirectional full multigrid methods with a full approximation scheme (FAS) in the nonlinear setting. Their parallelisation on the Cell hardware uses a temporal instead of a spatial decomposition, and processes operations in a vector-based manner. Memory latencies are reduced by a locality-preserving cache management and optimised access patterns. With images of size 316×252 pixels, we obtain dense flow fields for up to 210 frames per second.

Quantifying motion in video recordings of neonatal seizures by regularized optical flow methods

This paper presents the development of regularized optical flow computation methods and an evaluation of their performance in the extraction of quantitative motion information from video recordings of neonatal seizures. A general formulation of optical flow computation is presented and a mathematical framework for the development of practical tools for computing optical flow is outlined. In addition, this paper proposes an alternative formulation of the optical flow problem that relies on a discrete approximation of a family of quadratic functionals. These regularized optical flow computation methods are used to extract motion strength signals from video recordings of neonatal seizures.

Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods

Differential methods belong to the most widely used techniques for optic flow computation in image sequences. They can be classified into local methods such as the Lucas-Kanade technique or Bigun's structure tensor method, and into global methods such as the Horn/Schunck approach and its extensions. Often local methods are more robust under noise, while global techniques yield dense flow fields. The goal of this paper is to contribute to a better understanding and the design of novel differential methods in four ways: (i) We juxtapose the role of smoothing/regularisation processes that are required in local and global differential methods for optic flow computation. (ii) This discussion motivates us to describe and evaluate a novel method that combines important advantages of local and global approaches: It yields dense flow fields that are robust against noise. (iii) Spatiotemporal and nonlinear extensions as well as multiresolution frameworks are presented for this hybrid method. (iv) We propose a simple confidence measure for optic flow methods that minimise energy functionals. It allows to sparsify a dense flow field gradually, depending on the reliability required for the resulting flow. Comparisons with experiments from the literature demonstrate the favourable performance of the proposed methods and the confidence measure.

Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods

Differential methods belong to the most widely used techniques for optic flow computation in image sequences. They can be classified into local methods such as the Lucas–Kanade technique or Bigün's structure tensor method, and into global methods such as the Horn/Schunck approach and its extensions. Often local methods are more robust under noise, while global techniques yield dense flow fields. The goal of this paper is to contribute to a better understanding and the design of novel differential methods in four ways; (i) We juxtapose the role of smoothing/regularisation processes that are required in local and global differential methods for optic flow computation. (ii) This discussion motivates us to describe and evaluate a novel method that combines important advantages of local and global approaches: It yields dense flow fields that are robust against noise. (iii) Spatiotemporal and nonlinear extensions as well as multiresolution frameworks are presented for this hybrid method. (iv) We propose a simple confidence measure for optic flow methods that minimise energy functionals. It allows to sparsify a dense flow field gradually, depending on the reliability required for the resulting flow. Comparisons with experiments from the literature demonstrate the favourable performance of the proposed methods and the confidence measure.