Dense Motion Estimation Research Articles

Deep Optical Flow Supervised Learning With Prior Assumptions

Traditional methods for estimating optical flow use variational model that includes data term and smoothness term, which can build a constraint relationship between two adjacent images and optical flow. However, most of them are too slow to be used in real-time applications. Recently, convolutional neural networks have been used in optical flow area successfully. Many current learning methods use large data sets that contain ground truth for network training, which can make use of prior knowledge to estimate optical flow directly. However, these methods overemphasize the factor of deep learning and ignore advantages of many traditional assumptions used in variational framework for optical flow estimation. In this paper, inspired by classical energy-based optical flow methods, we propose a novel approach for dense motion estimation, which combines traditional prior assumptions with supervised learning network. During training, the variation in image brightness, gradient and spatial smoothness are embedded in network. Our method is tested on both synthetic and real scenes. The experimental results show that employing the prior assumptions during training can obtain more detailed and smoothed flow fields and can improve the accuracy of optical flow estimation.

Schlieren technique in soap film flows

We propose the use of the Schlieren technique as a tool to analyse the flows in soap film tunnels. The technique enables to visualize perturbations of the film produced by the interposition of an object in the flow. The variations of intensity of the image are produced as a consequence of the deviations of the light beam traversing the deformed surfaces of the film. The quality of the Schlieren image is compared to images produced by the conventional interferometric technique. The analysis of Schlieren images of a cylinder wake flow indicates that this technique enables an easy visualization of vortex centers. Post-processing of series of two successive images of a grid turbulent flow with a dense motion estimator is used to derive the velocity fields. The results obtained with this self-seeded flow show good agreement with the statistical properties of the 2D turbulent flows reported on the literature.

EasyFlow: increasing the convergence basin of variational image matching with a feature‐based cost

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non‐rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature‐based matching. The feature‐based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second‐order regularisation with external and self‐occlusion reasoning. Their framework achieves state‐of‐the‐art performance in several cases (standard optical flow benchmarks, wide‐baseline stereo and non‐rigid surface registration). Their framework has a modular design that customises to specific application needs.

Real-time target tracking of soft tissues in 3D ultrasound images based on robust visual information and mechanical simulation.

In this paper, we present a real-time approach that allows tracking deformable structures in 3D ultrasound sequences. Our method consists in obtaining the target displacements by combining robust dense motion estimation and mechanical model simulation. We perform evaluation of our method through simulated data, phantom data, and real-data. Results demonstrate that this novel approach has the advantage of providing correct motion estimation regarding different ultrasound shortcomings including speckle noise, large shadows and ultrasound gain variation. Furthermore, we show the good performance of our method with respect to state-of-the-art techniques by testing on the 3D databases provided by MICCAI CLUST'14 and CLUST'15 challenges.

Multi-reference combinatorial strategy towards longer long-term dense motion estimation

This paper addresses the estimation of accurate long-term dense motion fields from videos of complex scenes. With computer vision applications such as video editing in mind, we exploit optical flows estimated with various inter-frame distances and combine them through multi-step integration and statistical selection (MISS). In this context, managing numerous combinations of multi-step optical flows requires a complexity reduction scheme to overcome computational and memory issues. Our contribution is two-fold. First, we provide an exhaustive analysis of available single-reference complexity reduction strategies. Second, we propose a simple and efficient alternative related to multi-reference frames multi-step integration and statistical selection (MR-MISS). Our method automatically inserts intermediate reference frames once matching failures are detected to re-generate the motion estimation process and re-correlates the resulting dense trajectories. By this way, it reaches longer accurate displacement fields while efficiently reducing the complexity. Experiments on challenging sequences reveal improved results compared to state-of-the-art methods including existing MISS schemes both in terms of complexity reduction and accuracy improvement.

Counting-Based Particle Flux Estimation for Traffic Analysis in Live Cell Imaging

A quantitative analysis of the dynamic contents in fluorescence time-lapse microscopy is crucial to decipher the molecular mechanisms involved in cell functions. In this paper, we propose an original traffic analysis approach based on the counting of particles from frame to frame. The suggested method lies between individual object tracking and dense motion estimation (i.e., optical flow). Instead of tracking each moving particle, we estimate fluxes of particles between predefined and adjacent regions. The problem is formulated as the minimization of a global cost function and the approach allows us to process image sequences with a high number of particles and a high rate of particle appearances and disappearances. We propose to study the influence of object density, image partition scale, motion amplitude, and particle appearances/disappearances in a large variety of simulations. The potential of the method is finally demonstrated on real image sequences showing GFP-tagged Rab6 trafficking in confocal microscopy.

Foreground–Background Separation From Video Clips via Motion-Assisted Matrix Restoration

Separation of video clips into foreground and background components is a useful and important technique, making recognition, classification, and scene analysis more efficient. In this paper, we propose a motion-assisted matrix restoration (MAMR) model for foreground–background separation in video clips. In the proposed MAMR model, the backgrounds across frames are modeled by a low-rank matrix, while the foreground objects are modeled by a sparse matrix. To facilitate efficient foreground-background separation, a dense motion field is estimated for each frame, and mapped into a weighting matrix which indicates the likelihood that each pixel belongs to the background. Anchor frames are selected in the dense motion estimation to overcome the difficulty of detecting slowly moving objects and camouflages. In addition, we extend our model to a robust MAMR model against noise for practical applications. Evaluations on challenging datasets demonstrate that our method outperforms many other state-of-the-art methods, and is versatile for a wide range of surveillance videos.

Video Extruder: a semi-dense point tracker for extracting beams of trajectories in real time

Two crucial aspects of general-purpose embedded visual point tracking are addressed in this paper. First, the algorithm should reliably track as many points as possible. Second, the computation should achieve real-time video processing, which is challenging on low power embedded platforms. We propose a new multi-scale semi-dense point tracker called Video Extruder, whose purpose is to fill the gap between short-term, dense motion estimation (optical flow) and long-term, sparse salient point tracking. This paper presents a new detector, including a new salience function with low computational complexity and a new selection strategy that allows to obtain a large number of keypoints. Its density and reliability in mobile video scenarios are compared with those of the FAST detector. Then, a multi-scale matching strategy is presented, based on hybrid regional coarse-to-fine and temporal prediction, which provides robustness to large camera and object accelerations. Filtering and merging strategies are then used to eliminate most of the wrong or useless trajectories. Thanks to its high degree of parallelism, the proposed algorithm extracts beams of trajectories from the video very efficiently. We compare it with the state-of-the-art pyramidal Lucas---Kanade point tracker and show that, in short range mobile video scenarios, it yields similar quality results, while being up to one order of magnitude faster. Three different parallel implementations of this tracker are presented, on multi-core CPU, GPU and ARM SoCs. On a commodity 2010 CPU, it can track 8,500 points in a 640 × 480 video at 150 Hz.

Nonlinear Transform for Robust Dense Block-Based Motion Estimation

We present a noniterative multiresolution motion estimation strategy, involving block-based comparisons in each detail band of a Laplacian pyramid. A novel matching score is developed and analyzed. The proposed matching score is based on a class of nonlinear transformations of Laplacian detail bands, yielding 1-bit or 2-bit representations. The matching score is evaluated in a dense full-search motion estimation setting, with synthetic video frames and an optical flow data set. Together with a strategy for combining the matching scores across resolutions, the proposed method is shown to produce smoother and more robust estimates than mean square error (MSE) in each detail band and combined. It tolerates more of nontranslational motion, such as rotation, validating the analysis, while providing much better localization of the motion discontinuities. We also provide an efficient implementation of the motion estimation strategy and show that the computational complexity of the approach is closely related to the traditional MSE block-based full-search motion estimation procedure.

Compass Rose: A Rotational Robust Signature for Optical Flow Computation

This paper proposes a new image signature, called Compass Rose, which is particularly suited to optical flow computation. It is differentiable, fast to compute, and robust to additive illumination changes, translation, and fast rotation. We design a sparse flow computation system based on the invariance of the Compass Rose signatures. This is then extended to dense motion estimation by the addition of an optional diffusion step. Quantitative testing on several benchmark sequences shows the Compass Rose attains higher accuracy than the traditional flow signatures under a range of conditions. Finally, we demonstrate its application to human motion estimation, which is challenging for optical flow methods due to fast limb rotation.

Joint edge detection and motion estimation of cardiac MR image sequence by a phase field method

In this paper a variational framework for joint segmentation and motion estimation is employed for inspecting heart in Cine MRI sequences. A functional including Mumford–Shah segmentation and optical flow based dense motion estimation is approximated using the phase-field technique. The minimizer of the functional provides an optimum motion field and edge set by considering both spatial and temporal discontinuities. Exploiting calculus of variation principles, multiple partial differential equations associated with the Euler–Lagrange equations of the functional are extracted, first. Next, the finite element method is used to discretize the resulting PDEs for numerical solution. Several simulation runs are used to test the convergence and the parameter sensitivity of the method. It is further applied to a comprehensive set of clinical data in order to compare with conventional cascade methods. Developmental constraints are identified as memory usage and computational complexities, which may be resolved utilizing sparse matrix manipulations and similar techniques. Based on the results of this study, joint segmentation and motion estimation outperforms previously reported cascade approaches especially in segmentation. Experimental results substantiated that the proposed method extracts the motion field and the edge set more precisely in comparison with conventional cascade approaches. This superior result is the consequence of simultaneously considering the discontinuity in both motion field and image space and including consequent frames (usually five) in our joint process functional.

Three-Dimensional Motion Estimation of Atmospheric Layers From Image Sequences

In this paper, we address the problem of estimating 3-D motions of a stratified atmosphere from satellite image sequences. The analysis of 3-D atmospheric fluid flows associated with incomplete observation of atmospheric layers due to the sparsity of cloud systems is very difficult. This makes the estimation of dense atmospheric motion field from satellite image sequences very difficult. The recovery of the vertical component of fluid motion from a monocular sequence of image observations is a very challenging problem for which no solution exists in the literature. Based on a physically sound vertical decomposition of the atmosphere into cloud layers of different altitudes, we propose here a dense motion estimator dedicated to the extraction of 3-D wind fields characterizing the dynamics of a layered atmosphere. Wind estimation is performed over the complete 3-D space, using a multilayer model describing a stack of dynamic horizontal layers of evolving thickness, interacting at their boundaries via vertical winds. The efficiency of our approach is demonstrated on synthetic and real sequences.

Performance Improvement of Super-Resolution by Dense Motion Estimation Using Overlapped Block Matching

Performance Improvement of Super-Resolution by Dense Motion Estimation Using Overlapped Block Matching

Geodesic active regions and level set methods for motion estimation and tracking

Motion analysis in computer vision is a well-studied problem with numerous applications. In particular, the tasks of optical flow estimation and tracking are of increasing interest. In this paper, we propose a level set approach to address both aspects of motion analysis. Our approach relies on the propagation of smooth interfaces to perform tracking while using an incremental estimation of the motion models. Implicit representations are used to represent moving objects, and capture their motion parameters. Information from different sources like a boundary attraction term, a background subtraction component and a visual consistency constraint are considered. The Euler–Lagrange equations within a gradient descent method lead to a flow that deforms a set of initial curve towards the object boundaries as well an incremental robust estimator of their apparent motion. Partial extension of the proposed framework to address dense motion estimation and the case of moving observer is also presented. Promising results demonstrate the performance of the method.

One-dimensional dense disparity estimation for three-dimensional reconstruction

We present a method for fully automatic three-dimensional (3D) reconstruction from a pair of weakly calibrated images in order to deal with the modeling of complex rigid scenes. A two-dimensional (2D) triangular mesh model of the scene is calculated using a two-step algorithm mixing sparse matching and dense motion estimation approaches. The 2D mesh is iteratively refined to fit any arbitrary 3D surface. At convergence, each triangular patch corresponds to the projection of a 3D plane. The proposed algorithm relies first on a dense disparity field. The dense field estimation modelized within a robust framework is constrained by the epipolar geometry. The resulting field is then segmented according to homographic models using iterative Delaunay triangulation. In association with a weak calibration and camera motion estimation algorithm, this 2D planar model is used to obtain a VRML-compatible 3D model of the scene.

ON THE STUDY OF VLSI DERIVATION FOR OPTICAL FLOW ESTIMATION

In this paper we propose studying several ways to implement a realistic and efficient VLSI design for a gradient-based dense motion estimator. The kind of estimator we focus on belongs to the class of differential methods. It is classically based on the optical flow constraint equation in association with a smoothness regularization term and also incorporates robust cost functions to alleviate the influence of large residuals. This estimator is expressed as the minimization of a global energy function defined within the framework of an incremental formulation associated with a multiresolution setup. In order to make possible the conception of efficient hardware, we consider a modified minimization strategy. This new minimization strategy is not only well suited to VLSI derivation, it is also very efficient in terms of quality of the result. The complete VLSI derivation is realized using high-level specifications.

Left ventricular motion reconstruction based on elastic vector splines.

In medical imaging it is common to reconstruct dense motion estimates, from sparse measurements of that motion, using some form of elastic spline (thin-plate spline, snakes and other deformable models, etc.). Usually the elastic spline uses only bending energy (second-order smoothness constraint) or stretching energy (first-order smoothness constraint), or a combination of the two. These elastic splines belong to a family of elastic vector splines called the Laplacian splines. This spline family is derived from an energy minimization functional, which is composed of multiple-order smoothness constraints. These splines can be explicitly tuned to vary the smoothness of the solution according to the deformation in the modeled material/tissue. In this context, it is natural to question which members of the family will reconstruct the motion more accurately. We compare different members of this spline family to assess how well these splines reconstruct human cardiac motion. We find that the commonly used splines (containing first-order and/or second-order smoothness terms only) are not the most accurate for modeling human cardiac motion.