Optical Flow Formulation Research Articles

An accurate optical flow estimation of PIV using fluid velocity decomposition

In this study, we propose a novel optical flow formulation for estimating high-accuracy velocity fields from tracer particle image sequences. According to the Helmholtz velocity decomposition theorem, the proposed optical flow method decomposes the two-dimensional velocity field into four components: translation motion, linear distortion motion, shear distortion motion and rotation motion. In this context, regularization terms for different motion components are designed, which have a reasonable physical interpretation for the flow characteristics of the fluid. Subsequently, we design specific regularization parameters for the corresponding regularization terms according to the flow characteristics of the motion components. These regularization parameters can be adaptively adjusted with changes in the image space and velocity field. In addition, the data term of the optical flow formulation is based on the projected-motion equation derived from the continuity equation, which maintains the compressibility of the fluid in the two-dimensional plane. Velocity fields are estimated from synthetic tracer particle images and hypersonic experimental image sequences, and the velocity results are compared to those of an advanced cross-correlation-based PIV method and previous advanced optical flow methods. The results and comparisons indicate that the proposed method shows good performance and high measurement accuracy when acquiring compressible flow structures from fluid measurements.

A field-segmentation-based variational optical flow method for PIV measurements of nonuniform flows

Air flow fields have nonuniform velocity distributions because of the compressibility of air. Current optical flow-based particle image velocimetry (PIV) methods cannot thoroughly solve these distributions because of the global optical flow formulation. In this study, we propose a novel field-segmentation-based variational optical flow (FS-VOF) method for preserving the spatially discontinuous characteristics of nonuniform flow fields. The proposed method is designed based on the level-set segmentation method, in which the particle image is segmented according to the velocity distribution of the fluid flow. Subsequently, a segmented smoothness constraint term built from the assumption that velocity varies continuously in each segmented particle image region and the discontinuous structures of the velocity field are preserved at the boundaries of the segmented regions. In addition, the data term of the proposed method is based on the segmented regions and derived from the physics-based optical flow equation. The proposed method is evaluated over synthetic and experimental images, and the comparison with advanced variational optical flow algorithms demonstrates the effectiveness of our method in preserving the spatially discontinuous characteristics of nonuniform flow fields.

Depth from a Motion Algorithm and a Hardware Architecture for Smart Cameras.

Applications such as autonomous navigation, robot vision, and autonomous flying require depth map information of a scene. Depth can be estimated by using a single moving camera (depth from motion). However, the traditional depth from motion algorithms have low processing speeds and high hardware requirements that limit the embedded capabilities. In this work, we propose a hardware architecture for depth from motion that consists of a flow/depth transformation and a new optical flow algorithm. Our optical flow formulation consists in an extension of the stereo matching problem. A pixel-parallel/window-parallel approach where a correlation function based on the sum of absolute difference (SAD) computes the optical flow is proposed. Further, in order to improve the SAD, the curl of the intensity gradient as a preprocessing step is proposed. Experimental results demonstrated that it is possible to reach higher accuracy (90% of accuracy) compared with previous Field Programmable Gate Array (FPGA)-based optical flow algorithms. For the depth estimation, our algorithm delivers dense maps with motion and depth information on all image pixels, with a processing speed up to 128 times faster than that of previous work, making it possible to achieve high performance in the context of embedded applications.

Optical flow for image-based river velocity estimation

We present a novel motion estimation technique for image-based river velocimetry. It is based on the so-called optical flow, which is a well developed method for rigid motion estimation in image sequences, devised in computer vision community. Contrary to PIV (Particle Image Velocimetry) techniques, optical flow formulation is flexible enough to incorporate physics equations that govern the observed quantity motion. Over the past years, it has been adopted by experimental fluid dynamics community where many new models were introduced to better represent different fluids motions, (see [18] for a review). Our optical flow is based on the scalar transport equation and is augmented with a weighted diffusion term to compensate for small scale (non-captured) contributions. Additionally, since there is no ground truth data for such type of image sequences, we present a new evaluation method to assess the results. It is based on trajectory reconstruction of few Lagrangian particles of interest and a direct comparison against their manually-reconstructed trajectories. The new motion estimation technique outperformed traditional optical flow and PIV-based methods.

Bayesian optical flow with uncertainty quantification

Optical flow refers to the visual motion observed between two consecutive images. Since the degree of freedom is typically much larger than the constraints imposed by the image observations, the straightforward formulation of optical flow as an inverse problem is ill-posed. Standard approaches to determine optical flow rely on formulating and solving an optimization problem that contains both a data fidelity term and a regularization term, the latter effectively resolves the otherwise ill-posedness of the inverse problem. In this work, we depart from the deterministic formalism, and instead treat optical flow as a statistical inverse problem. We discuss how a classical optical flow solution can be interpreted as a point estimate in this more general framework. The statistical approach, whose ‘solution’ is a distribution of flow fields, which we refer to as Bayesian optical flow, allows not only ‘point’ estimates (e.g. the computation of average flow field), but also statistical estimates (e.g. quantification of uncertainty) that are beyond any standard method for optical flow. As application, we benchmark Bayesian optical flow together with uncertainty quantification using several types of prescribed ground-truth flow fields and images.

Motion estimation under location uncertainty for turbulent fluid flows

In this paper, we propose a novel optical flow formulation for estimating two-dimensional velocity fields from an image sequence depicting the evolution of a passive scalar transported by a fluid flow. This motion estimator relies on a stochastic representation of the flow allowing to incorporate naturally a notion of uncertainty in the flow measurement. In this context, the Eulerian fluid flow velocity field is decomposed into two components: a large-scale motion field and a small-scale uncertainty component. We define the small-scale component as a random field. Subsequently , the data term of the optical flow formulation is based on a stochastic transport equation, derived from the formalism under location uncertainty proposed in Memin (2014) and Resseguier et al. (2017a). In addition, a specific regulariza-tion term built from the assumption of constant kinetic energy involves the very same diffusion tensor as the one appearing in the data transport term. Opposite to the classical motion estimators, this enables us to devise an optical flow method dedicated to fluid flows in which the regularization parameter has now a clear physical interpretation and can be easily estimated. Experimental evaluations are presented on both synthetic and real world image sequences. Results and comparisons indicate very good performance of the proposed formulation for turbulent flow motion estimation.

Robust universal nonrigid motion correction framework for first-pass cardiac MR perfusion imaging.

To present and assess an automatic nonrigid image registration framework that compensates motion in cardiac magnetic resonance imaging (MRI) perfusion series and auxiliary images acquired under a wide range of conditions to facilitate myocardial perfusion quantification. Our framework combines discrete feature matching for large displacement estimation with a dense variational optical flow formulation in a multithreaded architecture. This framework was evaluated on 291 clinical subjects to register 1.5T and 3.0T steady-state free-precession (FISP) and fast low-angle shot (FLASH) dynamic contrast myocardial perfusion images, arterial input function (AIF) images, and proton density (PD)-weighted images acquired under breath-hold (BH) and free-breath (FB) settings. Our method significantly improved frame-to-frame appearance consistency compared to raw series, expressed in correlation coefficient (R2 = 0.996 ± 3.735E-3 vs. 0.978 ± 2.024E-2, P < 0.0001) and mutual information (3.823 ± 4.098E-1 vs. 2.967 ± 4.697E-1, P < 0.0001). It is applicable to both BH (R2 = 0.998 ± 3.217E-3 vs. 0.990 ± 7.527E-3) and FB (R2 = 0.995 ± 3.410E-3 vs. 0.968 ± 2.257E-3) paradigms as well as FISP and FLASH sequences. The method registers PD images to perfusion T1 series (9.70% max increase in R2 vs. no registration, P < 0.001) and also corrects motion in low-resolution AIF series (R2 = 0.987 ± 1.180E-2 vs. 0.964 ± 3.860E-2, P < 0.001). Finally, we showed the myocardial perfusion contrast dynamic was preserved in the motion-corrected images compared to the raw series (R2 = 0.995 ± 6.420E-3). The critical step of motion correction prior to pixel-wise cardiac MR perfusion quantification can be performed with the proposed universal system. It is applicable to a wide range of perfusion series and auxiliary images with different acquisition settings. 3 Technical Efficacy: Stage 1 J. Magn. Reson. Imaging 2017;46:1060-1072.

Monocular Concurrent Recovery of Structure and Motion Scene Flow

This paper describes a variational method of joint three-dimensional structure and motion scene flow recovery from a single image sequence. A basic scheme is developed by minimizing a functional with a term of conformity of scene flow and depth to the image sequence spatiotemporal variations, and quadratic smoothness regularization terms. The data term follows by re-writing optical velocity in the optical flow gradient constraint in terms of scene flow and depth. As a result, this problem statement is analogous to the classical Horn and Schunck optical flow formulation except that it involves scene flow and depth rather than image motion. When discretized, the Euler-Lagrange equations give a large scale sparse system of linear equations in the unknowns of the scene flow three coordinates and depth. The equations can be ordered in such a way that its matrix is symmetric positive definite such that they can be solved efficiently by Gauss-Seidel iterations. Experiments are shown to verify the scheme’s validity and efficiency.

Optical flow for incompressible turbulence motion estimation

We propose in this paper a new formulation of optical flow dedicated to 2D incompressible turbulent flows. It consists in minimizing an objective function constituted by an observation term and a regularization one. The observation term is based on the transport equation of the passive scalar field. For non-fully resolved scalar images, we propose to use the mixed model in large eddy simulation to determine the interaction between large scales and unresolved ones. The regularization term is based on the continuity equation of 2D incompressible flows. Compared to prototypical method, this regularizer preserves more vortex structures by eliminating constraints over the vorticity field. The evaluation of the proposed formulation is done over synthetic and experimental images, and the improvements in term of estimation are discussed.

Speed and Texture: An Empirical Study on Optical-Flow Accuracy in ADAS Scenarios

Increasing mobility in everyday life has led to the concern for the safety of automotives and human life. Computer vision has become a valuable tool for developing driver assistance applications that target such a concern. Many such vision-based assisting systems rely on motion estimation, where optical flow has shown its potential. A variational formulation of optical flow that achieves a dense flow field involves a data term and regularization terms. Depending on the image sequence, the regularization has to appropriately be weighted for better accuracy of the flow field. Because a vehicle can be driven in different kinds of environments, roads, and speeds, optical-flow estimation has to be accurately computed in all such scenarios. In this paper, we first present the polar representation of optical flow, which is quite suitable for driving scenarios due to the possibility that it offers to independently update regularization factors in different directional components. Then, we study the influence of vehicle speed and scene texture on optical-flow accuracy. Furthermore, we analyze the relationships of these specific characteristics on a driving scenario (vehicle speed and road texture) with the regularization weights in optical flow for better accuracy. As required by the work in this paper, we have generated several synthetic sequences along with ground-truth flow fields.

Revisiting Lucas-Kanade and Horn-Schunck

This paper revisits classical Lucas-Kanade (LK) and Horn-Schunck (HS) optical flow techniques. The aim is to provide a baseline for other researchers on these two timeless techniques. The formulations presented incorporate modern practices, namely multichannel, multi-resolution with refinement procedure (warping), non-quadratic penalisers and non-linear formulation of the brightness assumption. The experiments conducted demonstrate the performance enhancement that can be assigned to each modern practice. Thereby, the performance of the LK and HS is renewed in order to enable a fair comparison to other state-of-art techniques. The good performance of today's optical flow formulations is possible due to several modern practices, namely multi- resolution (coarse-to-fine) refinement that deals with large displacements, iterative methods based on warping interpolations, multi-channels, robustification of each channel (non- quadratic penalisers), gradient constancy assumption to handle small variation of the brightness value, and photometric colour spaces. Usually, the performance of state-of-art techniques is compared to the classical Lucas-Kanade (LK) and Horn-Schunck (HS) presented by Barron et al. (1994) (2) . We consider these comparisons unfair because the LK and HS do not contemplate the most important modern practices (such as, multi-resolution with refinement, nonlinear formulation of the brightness constancy assumption). Therefore, the full potential of both techniques is not considered. For that reason, this article intends to provide a modern formulation of the LK and HS that incorporates multi-channel, multi-resolution with refinement (iterative procedure based on warping) and nonlinear brightness assumption. In addition, non-quadratic penaliser based on the Charbonnier error function is considered for the neighbourhood weighting of the LK. Thereby, the contributions of this paper include:  Modern and coloured formulation of the LK and HS  Extensive qualitative and quantitative evaluation  Baseline performance analysis for other research The experimental analysis includes comparisons of both non-hierarchical and hierarchical pyramidal architectures. The contribution of the iterative refinement and multi-channel approach is also studied. Finally, the neighbourhood weights of the LK are iteratively defined using the non-quadratic penaliser. When implemented with appropriate and modern practices, classical optical flow formulations such as Lucas- Kanade (9) and Horn-Schunck (10) can achieve respectable optical flow estimations.

A Stream Function Approach to Optical Flow with Applications to Fluid Transport Dynamics

AbstractOptical flow is one of the classical problems in computer vision, but it has recently also been adapted to applications from other fields, such as fluid mechanics and dynamical systems. If the goal is to analyze the dynamics of system whose evolution is governed by a flow field that is the gradient of a potential function – which describes many flows in fluid dynamics – it is natural to approach the optical flow problem by reconstructing the potential function, also called the stream function, rather than reconstructing the components of the flow directly. This alternate approach allows one to impose scientific priors, via regularization, directly on the flow itself rather than on its components independently. We demonstrate the stream function formulation of optical flow and its application to reconstructing an oceanic fluid flow driven by satellite measurements. It is also shown how these flow fields can be used to analyze mixing and mass transport in the fluid system being imaged. (© 2011 Wiley‐VCH Verlag GmbH &amp; Co. KGaA, Weinheim)

Subgrid scale formulation of optical flow for the study of turbulent flow

We propose a new formulation of optical flow in the case of passive scalar or solid particles spreading in turbulent flows. The flow equation is defined from the scalar transport equation and a decomposition of our physical quantities (velocity and concentration fields) into two contributions, large and small scales, to account for the lack of spatial resolution in processed images. A subgrid scale model is introduced to model the small scale contribution. Comparisons are made with existing optical flow methods and Particle Image Velocimetry on synthetic and real sequences. The improvement of the estimation of velocity field by the proposed formulation is discussed in the case of a scalar turbulent propagation.

Left Ventricular Border Tracking Using Cardiac Motion Models and Optical Flow

The use of automated methods is becoming increasingly important for assessing cardiac function quantitatively and objectively. In this study, we propose a method for tracking three-dimensional (3-D) left ventricular contours. The method consists of a local optical flow tracker and a global tracker, which uses a statistical model of cardiac motion in an optical-flow formulation. We propose a combination of local and global trackers using gradient-based weights. The algorithm was tested on 35 echocardiographic sequences, with good results (surface error: 1.35 ± 0.46 mm, absolute volume error: 5.4 ± 4.8 mL). This demonstrates the method’s potential in automated tracking in clinical quality echocardiograms, facilitating the quantitative and objective assessment of cardiac function. (E-mail: k.leung@erasmusmc.nl)

Optical Flow for Verification

Abstract Modern numerical weather prediction (NWP) models produce forecasts that are gridded spatial fields. Digital images can also be viewed as gridded spatial fields, and as such, techniques from image analysis can be employed to address the problem of verification of NWP forecasts. One technique for estimating how images change temporally is called optical flow, where it is assumed that temporal changes in images (e.g., in a video) can be represented as a fluid flowing in some manner. Multiple realizations of the general idea have already been employed in verification problems as well as in data assimilation. Here, a specific formulation of optical flow, called Lucas–Kanade, is reviewed and generalized as a tool for estimating three components of forecast error: intensity and two components of displacement, direction and distance. The method is illustrated first on simulated data, and then on a 418-day series of 24-h forecasts of sea level pressure from one member [the Global Forecast System (GFS)–fifth-generation Pennsylvania State University–National Center for Atmospheric Research Mesoscale Model (MM5)] of the University of Washington’s Mesoscale Ensemble system. The simulation study confirms (and quantifies) the expectation that the method correctly assesses forecast errors. The method is also applied to a real dataset consisting of 418 twenty-four-hour forecasts spanning 2 April 2008–2 November 2009, demonstrating its value for analyzing NWP model performance. Results reveal a significant intensity bias in the subtropics, especially in the southern California region. They also expose a systematic east-northeast or downstream bias of approximately 50 km over land, possibly due to the treatment of terrain in the coarse-resolution model.

Image Similarity Based on Intensity Scaling

An image similarity measure based upon intensity scaling is investigated for the registration of images which differ in contrast. The measure is introduced for both optical flow and for finite displacements, and registration by the optical flow formulation is found to be independent of image order. While other morphological image matching methods involve differential formulations, the present work simply involves composition with scaling functions. Such rescaling is additionally found to have a smoothing effect on noisy images. Results obtained by Tikhonov regularization of scaling functions are superior to those obtained by a restricted set of basis functions. Computational results are performed for simple test cases as well as for realistic examples taken from magnetic resonance imaging. It is found that the computational cost for the implementation of scaling functions is very small, and yet the resulting image similarity can provide a match between images with smaller errors than that obtained by a sum of squared differences alone. In particular, images which are morphologically equivalent but which are scaled differently and even have different noise levels can be well matched with the scaling functions approach while the sum of squared differences would be unsuitable. In addition, matching is improved for image pairs in which contrast agent or an intensity modulation appears in one image and not in the other.

Robust rigid registration of retinal angiograms through optimization

Retinal fundus photographs are employed as standard diagnostic tools in ophthalmology. Serial photographs of the flow of fluorescein and indocyanine green (ICG) dye are used to determine the areas of the retinal lesions. For objective measurements of features, the registration of the images is a necessity. In this paper, we employ optimization techniques for registration with the help of 2-parameter translational motion model of retinal angiograms, based on non-linear pre-processing (Wiener filtering and morphological gradient) and computation of the similarity criteria for the alignment of the two gradient images for any given rigid transformation. The optimization methods are effectively employed to minimize the similarity criterion. The presence of noise, the variations in the background and the temporal variation of the fluorescence level pose serious problems in obtaining a robust registration of the retinal images. Moreover, local search strategies are not robust in the case of ICG angiograms, even if one uses a multiresolution approach. The present work makes a systematic comparison of different optimization techniques, namely the minimization method derived from the optical flow formulation, the Nelder-Mead local search and the HCIAC ant colony metaheuristic, each optimizing a similarity criterion for the gradient images. The impact of the resolution and median filtering of gradient image is studied and the robustness of the approaches is tested through experimental studies, performed on macular fluorescein and ICG angiographies. Our proposed optimization techniques have shown interesting results especially for high resolution difficult registration problems. Moreover, this approach seems promising for affine (6-parameter motion model) or elastical registrations.

Medical Image Registration and Interpolation by Optical Flow with Maximal Rigidity

In this paper a variational method for registering or mapping like points in medical images is proposed and analyzed. The proposed variational principle penalizes a departure from rigidity and thereby provides a natural generalization of strictly rigid registration techniques used widely in medical contexts. Difficulties with finite displacements are elucidated, and alternative infinitesimal displacements are developed for an optical flow formulation which also permits image interpolation. The variational penalty against non-rigid flows provides sufficient regularization for a well-posed minimization and yet does not rule out irregular registrations corresponding to an object excision. Image similarity is measured by penalizing the variation of intensity along optical flow trajectories. The approach proposed here is also independent of the order in which images are taken. For computations, a lumped finite element Eulerian discretization is used to solve for the optical flow. Also, a Lagrangian integration of the intensity along optical flow trajectories has the advantage of prohibiting diffusion among trajectories which would otherwise blur interpolated images. The subtle aspects of the methods developed are illustrated in terms of simple examples, and the approach is finally applied to the registration of magnetic resonance images.

Variational optical flow computation in real time

This paper investigates the usefulness of bidirectional multigrid methods for variational optical flow computations. Although these numerical schemes are among the fastest methods for solving equation systems, they are rarely applied in the field of computer vision. We demonstrate how to employ those numerical methods for the treatment of variational optical flow formulations and show that the efficiency of this approach even allows for real-time performance on standard PCs. As a representative for variational optic flow methods, we consider the recently introduced combined local-global method. It can be considered as a noise-robust generalization of the Horn and Schunck technique. We present a decoupled, as well as a coupled, version of the classical Gauss-Seidel solver, and we develop several multgrid implementations based on a discretization coarse grid approximation. In contrast, with standard bidirectional multigrid algorithms, we take advantage of intergrid transfer operators that allow for nondyadic grid hierarchies. As a consequence, no restrictions concerning the image size or the number of traversed levels have to be imposed. In the experimental section, we juxtapose the developed multigrid schemes and demonstrate their superior performance when compared to unidirectional multgrid methods and nonhierachical solvers. For the well-known 316 x 252 Yosemite sequence, we succeeded in computing the complete set of dense flow fields in three quarters of a second on a 3.06-GHz Pentium4 PC. This corresponds to a frame rate of 18 flow fields per second which outperforms the widely-used Gauss-Seidel method by almost three orders of magnitude.

Direct recovering of Nth order surface structure using unified optical flow field

There are two different approaches for estimation of structure and/or motion of objects from image sequences in the computer vision community today. One is the optical flow approach, and the other is the feature correspondence approach. There are many difficulties and limitations encountered with the feature correspondence method, while the optical flow method requires a substantial amount of extra calculations if the optical flow is to be computed as an intermediate step. Direct methods have been developed, that use the optical flow approach, but avoid computing the full optical flow field as an intermediate step for recovering structure and motion. The unified optical flow field (UOFF) theory was recently established. It is an extension of the optical flow formulations to the temporal-spatial domain. In this paper, a direct method is developed to reconstruct the curved surface structure characterized by an N-degree polynomial equation based on the UOFF. It is a direct method since it also directly recovers surface structure, without computing the quantities in the UOFF. However, it is different from the direct method in that it is based on the theoretical framework of the unified spatial-and-temporal optical field instead of the optical flow field determined. Hence, the structure is recovered from a given pair of stereo images instead of from a monocular image sequence. Two computer simulations: a sphere which is a second-order surface and an unbounded α-shaped surface which is a third-order one are presented. The fairly good results demonstrate the effectiveness of our direct method using the UOFF. Some numerical consideration and error analysis are also included. Compared with the existing techniques of the direct method where only the first-order surface can be recovered, the new method has made significant progress in the recovery of curved structure.