Optical Flow Estimation Algorithm Research Articles

Optical flow estimation based on the structure–texture image decomposition

Optical flow approaches for motion estimation calculate vector fields which determine the apparent velocities of objects in time-varying image sequences. Image motion estimation is a fundamental issue in low-level vision and is used in many applications in image sequence processing, such as robot navigation, object tracking, image coding and structure reconstruction. The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. Actually, several methods are used to estimate the optical flow, but a good compromise between computational cost and accuracy is hard to achieve. This work presents a combined local–global total variation approach with structure–texture image decomposition. The combination is used to control the propagation phenomena and to gain robustness against illumination changes, influence of noise on the results and sensitivity to outliers. The resulted method is able to compute larger displacements in a reasonable time.

Blind motion deblurring using optical flow

In recent years, a lot of motion deblurring methods have been proposed, most of which divide the motion blur problem into two parts: camera shake and object motion, and just deal with only one side of the problem. As a result, in this paper we focus on seeking a effective model to handle these two kinds of motion blur simultaneously. An optical flow based model is then proposed. In particular, the blurred image can be formulated as an integration of some clear intermediate images after optical based transformed. Meanwhile an effective algorithm based on this new model is introduced to deblur different types of blurry images. In this work, we not only use the optical flow to represent the blur process but also derive the optical flow estimation algorithm to help estimate the non-uniform PSF for each pixel. In addition, we show that by incorporating different kinds of constraints our method can get more robust and stable results. The experiments on both synthesized and real images show that our algorithm can effectively recover the latent images and get comparable results with some state of the art algorithms.

ESTIMATION OF THE CROSS CORRELATION BASED OPTICAL FLOW FOR VIDEO SURVEILLANCE

This paper describes a new algorithm to calculate cross-correlation function. We combined box filtering technique for calculation of cross correlation coefficients with parallel processing using MMX/SSE technology of modern general purpose processors. We have used this algorithm for real time optical flow estimation between frames of video sequence. Our algorithm was tested on real world video sequences obtained from the cameras of video surveillance system.

Illumination Robust Video Foreground Prediction Based on Color Recovering

Video foreground prediction is a technique to estimate the probability of each pixel being foreground in current frame based on a foreground segmentation result of its previous frame. Existing foreground prediction algorithms usually assume that the illumination conditions are constant for consecutive frames. Therefore, they cannot predict foreground accurately when the illumination condition changes sharply between video frames. In this paper, a new robust video foreground prediction algorithm is proposed based on color recovering, which is derived based on an observation that the illumination changes are locally smooth. By integrating color recovering with an optical flow estimation algorithm and an opacity propagation algorithm, the negative impact of the illumination changes could be removed. Experimental results show that the proposed algorithm can get more accurate results for videos with illumination changes compared with the existing foreground prediction algorithms.

A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them

The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that “classical” flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. One key implementation detail is the median filtering of intermediate flow fields during optimization. While this improves the robustness of classical methods it actually leads to higher energy solutions, meaning that these methods are not optimizing the original objective function. To understand the principles behind this phenomenon, we derive a new objective function that formalizes the median filtering heuristic. This objective function includes a non-local smoothness term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that can better preserve motion details. To take advantage of the trend towards video in wide-screen format, we further introduce an asymmetric pyramid downsampling scheme that enables the estimation of longer range horizontal motions. The methods are evaluated on the Middlebury, MPI Sintel, and KITTI datasets using the same parameter settings.

An Improved Motion Image Optical Flow Algorithm

We proposed an improved motion image optical flow method which can artificially adjust and select the threshold value according to the reality condition and different position of the image .Using traditional method,such as Horn-Schunck’s optical flow estimation algorithm,will possibly lead to the problem of losing edge information or obtaining unsmooth optical flow field, when weight coefficient is too big or too small at the same time. The experimental results show that the motion image and its edge are more clear and the value of PSNR and RMSE have been all improved by using the improved algorithm, moreover the improved algorithm will not increase more computation complexity.

Design Methods for Parallel Hardware Implementation of Multimedia Iterative Algorithms

The methodology from this paper exploits fine and coarse-grained parallelism for the automated design of digital architectures for multimedia applications. Specific focus is placed on iterative algorithms, as demonstrated through a case study of the Chambolle algorithm for optical flow estimation.

Motion estimation of magnetic resonance cardiac images using the Wigner-Ville and hough transforms

Myocardial motion analysis and quantification is of utmost importance for analyzing contractile heart abnormalities and it can be a symptom of a coronary artery disease. A fundamental problem in processing sequences of images is the computation of the optical flow, which is an approximation of the real image motion. This paper presents a new algorithm for optical flow estimation based on a spatiotemporal-frequency (STF) approach. More specifically it relies on the computation of the Wigner-Ville distribution (WVD) and the Hough Transform (HT) of the motion sequences. The latter is a well-known line and shape detection method that is highly robust against incomplete data and noise. The rationale of using the HT in this context is that it provides a value of the displacement field from the STF representation. In addition, a probabilistic approach based on Gaussian mixtures has been implemented in order to improve the accuracy of the motion detection. Experimental results in the case of synthetic sequences are compared with an implementation of the variational technique for local and global motion estimation, where it is shown that the results are accurate and robust to noise degradations. Results obtained with real cardiac magnetic resonance images are presented.

A genetic algorithm for optical flow estimation

This paper illustrates a new optical flow estimation technique that builds upon a genetic algorithm (GA). First, the current frame is segmented into generic shape regions, using only luminance and color information. For each region, a two-parameter motion model is estimated using a GA. The fittest individuals identified at the end of this step are used to initialize the population of the second step of the algorithm, which estimates a six-parameter affine motion model, again using a GA. The proposed method is compared with a multi-resolution version of the well-known Lucas–Kanade differential algorithm. Our simulations demonstrate that, with respect to Lucas–Kanade, it significantly reduces the energy of the motion-compensated residual error.

Optical flow estimation using temporally oversampled video

Recent advances in imaging sensor technology make high frame-rate video capture practical. As demonstrated in previous work, this capability can be used to enhance the performance of many image and video processing applications. The idea is to use the high frame-rate capability to temporally oversample the scene and, thus, to obtain more accurate information about scene motion and illumination. This information is then used to improve the performance of image and standard frame-rate video applications. This paper investigates the use of temporal oversampling to improve the accuracy of optical flow estimation (OFE). A method for obtaining high accuracy optical flow estimates at a conventional standard frame rate, e.g., 30 frames/s, by first capturing and processing a high frame-rate version of the video is presented. The method uses the Lucas-Kanade algorithm to obtain optical flow estimates at a high frame rate, which are then accumulated and refined to estimate the optical flow at the desired standard frame rate. The method demonstrates significant improvements in OFE accuracy both on synthetically generated video sequences and on a real video sequence captured using an experimental high-speed imaging system. It is then shown that a key benefit of using temporal oversampling to estimate optical flow is the reduction in motion aliasing. Using sinusoidal input sequences, the reduction in motion aliasing is identified and the desired minimum sampling rate as a function of the velocity and spatial bandwidth of the scene is determined. Using both synthetic and real video sequences, it is shown that temporal oversampling improves OFE accuracy by reducing motion aliasing not only for areas with large displacements but also for areas with small displacements and high spatial frequencies. The use of other OFE algorithms with temporally oversampled video is then discussed. In particular, the Haussecker algorithm is extended to work with high frame-rate sequences. This extension demonstrates yet another important benefit of temporal oversampling, which is improving OFE accuracy when brightness varies with time.

Optical flow detection using segmentation and MDL criterion

AbstractKanatani applied the concept of noise to optical flow calculation using the constraint equation proposed by Horn, and presented a renormalization method which estimates the flow and the noise components simultaneously. In the renormalization method, the image is segmented into blocks, and the constraint that flows of the pixels in a block remain the same is assumed. This paper applies MDL criterion to the renormalization method and proposes an optical flow estimation algorithm that segments the image self‐organizationally. It is assumed that the sender has a pair of images from time‐ordered sequence and the receiver has one of these images. In this assumption, the image segmentation is determined by using MDL criterion so that the intensity gradient vectors generated from the image pair are encoded most efficiently. Each intensity gradient vector is encoded to two components, namely, the flow vector and the error. The image segmentation and the flow vectors obtained by this procedure are defined as the estimated optical flows. As an experiment, the algorithm is applied to artificial image pairs and a real image pair and the results are presented. © 2004 Wiley Periodicals, Inc. Syst Comp Jpn, 35(9): 51–59, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10335

Accurate dense optical flow estimation using adaptive structure tensors and a parametric model

An accurate optical flow estimation algorithm is proposed in this paper. By combining the three-dimensional (3D) structure tensor with a parametric flow model, the optical flow estimation problem is converted to a generalized eigenvalue problem. The optical flow can be accurately estimated from the generalized eigenvectors. The confidence measure derived from the generalized eigenvalues is used to adaptively adjust the coherent motion region to further improve the accuracy. Experiments using both synthetic sequences with ground truth and real sequences illustrate our method. Comparisons with classical and recently published methods are also given to demonstrate the accuracy of our algorithm.

Robust optical flow estimation based on a sparse motion trajectory set

This paper presents an approach to the problem of estimating a dense optical flow field. The approach is based on a multiframe, irregularly spaced motion trajectory set, where each trajectory describes the motion of a given point as a function of time. From this motion trajectory set a dense flow field is estimated using a process of interpolation. A set of localized motion models are estimated, with each pixel labeled as belonging to one of the motion models. A Markov random field framework is adopted, allowing the incorporation of contextual constraints to encourage region-like structures. The approach is compared with a number of conventional optical flow estimation algorithms taken over a number of real and synthetic sequences. Results indicate that the method produces more accurate results for sequences with known ground truth flow. Also, applying the method to real sequences with unknown flow results in lower DFD, for all of the sequences tested.

Internal Shearing within the Hearing Organ Evoked by Basilar Membrane Motion

The vibration of the hearing organ that occurs during sound stimulation is based on mechanical interactions between different cellular structures inside the organ of Corti. The exact nature of these interactions is unclear and subject to debate. In this study, dynamic structural changes were produced by stepwise alterations of scala tympani pressure in an in vitro preparation of the guinea pig temporal bone. Confocal images were acquired at each level of pressure. In this way, the motion of several structures could be observed simultaneously with high resolution in a nearly intact system. Images were analyzed using a novel wavelet-based optical flow estimation algorithm. Under these conditions, the reticular lamina moved as a stiff plate with a center of rotation in the region of the inner hair cells. Despite being enclosed in several types of supporting cells, the inner hair cells, together with the adjacent inner pillar cells, moved in a manner signifying high compliance. The outer hair cells displayed radial motion indicative of cellular bending. Together, these results show that shearing motion occurs between several parts of the organ, and that structural relationships within the organ change dynamically during displacement of the basilar membrane.

Wavelet-based optical flow estimation

A new algorithm for accurate optical flow (OF) estimation using discrete wavelet approximation is proposed. The computation of OF depends on minimizing the image and smoothness constraints. The proposed method takes advantages of the nature of wavelet theory, which can efficiently and accurately approximate any function. OF vectors and image functions are represented by means of linear combinations of scaling basis functions. Based on such wavelet-based approximation, the leading coefficients of these basis functions carry global information about the approximated functions. The proposed method can successfully convert the problem of minimizing a constraint function into that of solving a linear system of a quadratic and convex function of scaling coefficients. Once all the corresponding coefficients are determined, the flow vectors can be obtained accordingly. Experiments have been conducted on both synthetic and real image sequences. In terms of accuracy, the results show that our approach outperforms the existing methods which adopted the same objective function as ours.

A Two-Stage Algorithm for Motion Discontinuity-Preserving Optical Flow Estimation

This paper proposes a two-stage optical flow estimation algorithm for video sequences based on feature-based matching and module-fusion. In the first stage, the algorithm computes initial translation and rotation parameters using a relaxation scheme at feature points and object-oriented anisotropic propagation in continuous and discontinuous regions. In the second stage, iterative-improvement optical flow estimation based on the strongly coupled recurrent module-fusion and adaptive smoothing is introduced. Computer simulation results of the proposed and conventional algorithms are shown.

Real-time parallel computation of disparity and optical flow using phase difference

A phase-difference-based algorithm for disparity and optical flow estimation is implemented on a TI-C40-based parallel DSP system. The module performs real-time computation of disparity maps on images of size 128 × 128 pixels and computation of optical flows on images of size 64 × 64 pixels. This paper describes the algorithm and its parallel implementation. Processing times required for the computation of disparity maps and velocity fields and measures of the algorithm's performance are reported in detail.

A fast scalable algorithm for discontinuous optical flow estimation

Multiple moving objects, partially occluded objects, or even a single object moving against the background gives rise to discontinuities in the optical flow field in corresponding image sequences. While uniform global regularization based moderately fast techniques cannot provide accurate estimates of the discontinuous flow field, statistical optimization based accurate techniques suffer from excessive solution time. A 'weighted anisotropic' smoothness based numerically robust algorithm is proposed that can generate discontinuous optical flow field with high speed and linear computational complexity. Weighted sum of the first-order spatial derivatives of the flow field is used for regularization. Less regularization is performed where strong gradient information is available. The flow field at any point is interpolated more from those at neighboring points along the weaker intensity gradient component. Such intensity gradient weighted regularization leads to Euler-Lagrange equations with strong anisotropies coupled with discontinuities in their coefficients. A robust multilevel iterative technique, that recursively generates coarse-level problems based on intensity gradient weighted smoothing weights, is employed to estimate discontinuous optical flow field. Experimental results are presented to demonstrate the efficacy of the proposed technique.

Multiscale appoaches to moving target detection in image sequences

We discuss a novel multiscale approach to the detection of moving objects in a sequence of images. The approach is based in part on a multiframe generalization of an optical flow estimation algorithm previously developed by two of the authors. This algorithm provides an extremely efficient multiscale method for estimating optical flow in an image sequence and allows for the temporal accumulation of motion information. Moving objects in a sequence correspond to discontinuities in the true optical flow and, as a result, the residual image associated with the estimated optical flow can be used as a basis for detecting these objects. We propose an approach to detection based on morphological processing of the residual image, and illustrate its potential on real data.

Multiple optical flow—fundamental constraint equations and a unified computational theory for detecting motion transparency and motion boundaries

AbstractConventional optical flow estimation algorithms were based on fundamental constraint equations that assume the existence of a single‐flow vector at each point in the image. However, since these fundamental constraint equations are not satisfied in certain regions, correct flow estimation can become impossible. Such regions include those where multiple flows arise due to objects with reflective surfaces and transparency or in regions where the flows become discontinuous at the boundaries of an object.This paper derives first extensions of the conventional fundamental constraint equations for optical flow in the real and frequency domains that are satisfied when n flow vectors exist at each point in the image. Algorithms are also given for solving the fundamental constraint equations in the general n‐fold flow case. Then for the double‐flow case, an algorithm is derived by computing two sets of flow parameters in closed‐form equations from the time‐varying image Gabor expansion.Next, margin energy is defined as the index for representing how well the optimization problem is posed for each degree of multiplicity (“well‐posedness”), and a criterion is obtained to determine the number of flow vectors at each image point. Finally, the effectiveness of the algorithm is verified for synthesized images containing motion transparency and motion boundaries.This papers proposes fundamental constraint equations for multiple optical flow as the basic equations for extending the various conventional motion vision models and time‐varying image analysis algorithms so that they can handle multiple motion transparency, and motion boundaries.