Optical Flow Estimation Algorithm Research Articles

SSTM: Spatiotemporal recurrent transformers for multi-frame optical flow estimation

Inaccurate optical flow estimates in and near occluded regions, and out-of-boundary regions are two of the current significant limitations of optical flow estimation algorithms. Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence. While this approach gives good flow estimates, it fails to generalize optical flows in occluded regions mainly due to limited local evidence regarding moving elements in a scene. In this work, we propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences. Our underlying hypothesis is that by understanding temporal scene dynamics from longer sequences with more than two frames, we can characterize pixel-wise dependencies in a larger spatiotemporal domain, generalize complex motion patterns and thereby improve the accuracy of optical flow estimates in occluded regions. We present learning-based spatiotemporal recurrent transformers for multi-frame based optical flow estimation (SSTMs). Our method utilizes 3D Convolutional Gated Recurrent Units (3D-ConvGRUs) and spatiotemporal transformers to learn recurrent space–time motion dynamics and global dependencies in the scene and provide a generalized optical flow estimation. When compared with recent state-of-the-art two-frame and multi-frame methods on real world and synthetic datasets, performance of the SSTMs were significantly higher in occluded and out-of-boundary regions. Among all published state-of-the-art multi-frame methods, SSTM achieved state-of the-art results on the Sintel Final and KITTI2015 benchmark datasets. Software code, data and instructions: https://github.com/Computational-Ocularscience/SSTM.

Automated crack detection in laminated composites by optical flow measurements

In this paper, an automated crack detection method for laminated composites of carbon fiber reinforced polymers (CFRP) is presented. Image correlation is performed on micrographs of the unspeckled specimen edge using DeepFlow, an algorithm for optical flow estimation. Cracks are then detected from local maxima of the displacement gradient evaluated along the midline of the plies. The detection strategy is applied to 3 different materials, with increasingly irregular microstructures, and on quasi-isotropic layups (a challenging case for crack detection). The crack density evolution in the 90∘ and ±45∘ plies is presented. The proposed method proves its capacity to detect cracks, even in complex configurations where other methods fail, and can provide an extensive amount of data that are useful for the identification of damage models for laminates. The procedure is largely automated in order to make it suitable for industrial applications.

Optical Flow Estimation Algorithm of Event Sensor Based on Improved Plane Fitting

Optical Flow Estimation Algorithm of Event Sensor Based on Improved Plane Fitting

Subgrid Variational Optimized Optical Flow Estimation Algorithm for Image Velocimetry

The variational optical flow model is used in this work to investigate a subgrid-scale optimization approach for modeling complex fluid flows in image sequences and estimating their two-dimensional velocity fields. To solve the problem of lack of sub-grid small-scale structure information in variational optical flow estimation, we combine the motion laws of incompressible fluids. Introducing the idea of large eddy simulation, the instantaneous motion can be decomposed into large-scale motion and a small-scale turbulence in the data term. The Smagorinsky model is used to model and solve the small-scale turbulence. The improved subgrid scale Horn-Schunck (SGS-HS) optical flow algorithm provides better results in velocity field estimation of turbulent image sequences than the traditional Farneback dense optical flow algorithm. To make the SGS-HS algorithm equally competent for the open channel flow measurement task, a velocity gradient constraint is chosen for the canonical term of the model, which is used to improve the accuracy of the SGS-HS algorithm in velocimetric experiments in the case of the relatively uniform flow direction of the open channel flow field. The experimental results show that our algorithm has better performance in open channel velocimetry compared with the conventional algorithm.

Motion Video Image Based on Sensor and Multiprocessing Technology in Track and Field Sports

In order to use motion video image processing technology to better guide and evaluate athletes’ technical and tactical ability, this paper first constructed a motion video image acquisition system. Secondly, a fuzzy kernel estimation method for uniform linear motion based on cepstrum property is proposed. Then, the motion blur areas to be processed are selected, and only these areas are deblurred. Finally, the effective removal of local motion blur is realized, and a clear scene image is obtained. Then, the maximum value of the total entropy of the image is calculated by using the information entropy theory, and the particle swarm optimization algorithm is introduced to find the maximum threshold of image segmentation. Finally, small wave optical flow estimation algorithm and rectangular window scanning algorithm across scales of motion image target detection algorithm are proposed, to not only solve the traditional optical flow estimation for fast moving object detection accuracy but also improve the efficiency of the optical flow computation. Compared to many kinds of algorithms, this paper proposed an algorithm that can improve the accuracy of moving target detection and measurement accuracy. And, the detection accuracy of the proposed algorithm is up to 86.5%. The estimated accuracy was as high as 65%. The segmentation accuracy is up to 95%.

Generating High-Quality Panorama by View Synthesis Based on Optical Flow Estimation.

Generating high-quality panorama is a key element in promoting the development of VR content. The panoramas generated by the traditional image stitching algorithm have some limitations, such as artifacts and irregular shapes. We consider solving this problem from the perspective of view synthesis. We propose a view synthesis approach based on optical flow to generate a high-quality omnidirectional panorama. In the first stage, we present a novel optical flow estimation algorithm to establish a dense correspondence between the overlapping areas of the left and right views. The result obtained can be approximated as the parallax of the scene. In the second stage, the reconstructed version of the left and the right views is generated by warping the pixels under the guidance of optical flow, and the alpha blending algorithm is used to synthesize the final novel view. Experimental results demonstrate that the subjective experience obtained by our approach is better than the comparison algorithm without cracks or artifacts. Besides the commonly used image quality assessment PSNR and SSIM, we also calculate MP-PSNR, which can provide accurate high-quality predictions for synthesized views. Our approach can achieve an improvement of about 1 dB in MP-PSNR and PSNR and 25% in SSIM, respectively.

Low-Latency and Scene-Robust Optical Flow Stream and Angular Velocity Estimation

Event cameras are bio-inspired sensors that capture intensity changes of pixels individually, and generate asynchronous and independent “events”. Due to the fundamental difference from the conventional cameras, most research on event cameras builds a global event frame by grouping events according to their timestamps or their number to employ traditional computer vision algorithms. However, in order to take advantage of event cameras, it makes sense to generate asynchronous output on an event-by-event basis. In this paper, we propose an optical flow estimation algorithm with low latency and robustness to various scenes to utilize the advantage of the event camera by enhancing the existing optical flow algorithm. Furthermore, we estimate angular velocity with low latency using the proposed optical flow stream. For the validation of algorithms, we evaluate the accuracy and latency of optical flow with publicly available datasets. Moreover, we assess the performance of the proposed angular velocity estimation in comparison to the existing algorithms. Both validations suggest that our asynchronous optical flow shows comparable accuracy to the existing algorithms and the latency is reduced by half compared to the existing block matching algorithm on average. Also, our angular velocity estimation is superior to the existing algorithms in terms of accuracy and robustness while showing low latency within 15 ms consistently.

Quantitative Comparative Study of Video Denoising with Optical Flow Estimation in Spatial and Transform Domains

In this paper we establish an extensive quantitative comparative study of patch-based video denoising with optical flow estimation algorithms. Namely, SPTWO, VBM3D and VBM4D algorithms are considered. The aim of this study is to combine these video denoising algorithms in a hybrid proposed process to take advantage of. SPTWO takes advantage of the self-similarity and redundancy of adjacent frames. The proposed hybrid algorithm and the three video denoising algorithms are implemented and tested on real sequences degraded by various level Additive White Gaussian Noise (AWGN). The obtained results are compared in terms of the most used performance criteria for various test cases. The performance criteria computed in this study are: RMSE and SSIM in addition to the running time and visual quality of the sequence video. Experimental results, illustrate that the proposed algorithm and SPTWO provide the best video quality and appear to be efficient in terms of preserving fine texture and detail reconstruction.

A Filtering-Based Framework for Optical Flow Estimation

We present a novel optical flow estimation framework that reinterprets most popular algorithms (e.g., the Horn-Schunck model, PDE-based models, TV-based models, and nonlocal-based models) from an iterative filtering perspective. Different regularizers in the related optical flow estimation algorithms can be achieved by adopting different filtering operations. The key merit of the proposed framework is that a variety of filtering models and corresponding optimization strategies, which are used in image processing, can be directly utilized in the regularization term, leading to a convenient way of designing appropriate optical flow estimation algorithms. Under the proposed filtering framework, we demonstrate easily designing optical flow estimation algorithms with respect to translation consistency, rotation consistency, and divergence consistency constraints by adopting different spatial filters, respectively. We also derive a novel optical flow estimation algorithm with 3D filtering-based model for the regularization term, which makes use of nonlocal self-similarity and sparse characteristic of optical flow field. Benefited from the advantages of patch-based nonlocal sparse regularizer, the derived algorithm can remove outliers while preserving sharp flow edges and important motion details. At present, the proposed algorithm achieves a high ranking on the challenging MPI-Sintel dataset and shows good performance on the Kitti flow 2015 and Middlebury datasets.

Optical Flow Estimation with Occlusion Detection

The dense optical flow estimation under occlusion is a challenging task. Occlusion may result in ambiguity in optical flow estimation, while accurate occlusion detection can reduce the error. In this paper, we propose a robust optical flow estimation algorithm with reliable occlusion detection. Firstly, the occlusion areas in successive video frames are detected by integrating various information from multiple sources including feature matching, motion edges, warped images and occlusion consistency. Then optimization function with occlusion coefficient and selective region smoothing are used to obtain the optical flow estimation of the non-occlusion areas and occlusion areas respectively. Experimental results show that the algorithm proposed in this paper is an effective algorithm for dense optical flow estimation.

Robust optical flow estimation based on wavelet

Concentrating on the issue that the systematic error caused by variation of illumination conditions and sensor noise leads to poor robustness and low accuracy of optical flow calculation, based on the wavelet multi-resolution theory, a robust optical flow estimation method is developed in this paper. With the multi-resolution characteristics of wavelet, the systematic error is incorporated into the calculation of optical flow to improve the robustness and estimation accuracy. In what follows, the total least squares method can be exploited to solve the obtained overdetermined wavelet optical flow equations such that the optical flow vector can be achieved. As compared to the traditional Lucas–Kanade approach, Horn–Schunck method, and the optical flow estimation algorithm in omnidirectional images using wavelet approach, simulation results show that the proposed algorithm can significantly improve the accuracy of optical flow estimation and the robustness of the optical flow field.

Comparative Performance Analysis of Optical Flow Algorithms for Anomaly Detection

Presently there is major interest in visual surveillance systems for crowd anomaly detection. Horn-Schunck, Lucas–Kanade and Farneback optical flow methods are used for the estimation of motion in the scene. The motion vectors; magnitude and orientation are analyzed in this work to detect anomaly in crowd. It helps classify the crowd as normal or abnormal, walking or running, vehicle entering in crowd. Various databases are evaluated to check the validity of the feature extraction process. In order to cluster the behaviour as normal or abnormal, Artificial Neural Network is used as a classifier. The results from Farneback optical flow estimation algorithm are promising for crowd behaviour understanding and anomaly detection.

Context-adaptive matching for optical flow

Modern sparse-to-dense optical flow estimation algorithms usually achieve state-of-art performance. Those algorithms need two steps: matching and interpolation. Matching is often unreliable for very large displacement optical flow due to illumination changes, deformations and occlusion etc. Moreover, conspicuous errors around motion discontinuities still keep serious as most methods consider edge only at interpolation step. The context-adaptive matching (CAM) is proposed for optical flow which is better at large displacement and edge preserving. The CAM is selective in feature extraction, adaptive in flow propagation and search radius adjusting. Selective features are proposed to consider edge preserving in matching step. Except for the usually used SIFT descriptor, the local directional pattern flow (LDPF) is introduced to keep more edge structure, and the oriented fast and rotated brief (ORB) is utilized to select out several most similar candidates. Unlike coarse-to-fine matching, which proposed a propagation step with only neighbors, we propose adaptive propagation to extend the matching candidates in order to improve the possibility of getting right correspondences. Furthermore, guided by prior knowledge and taking advantage of upper layers results, adaptive radius instead of constrained radius are proposed at finer layers. The CAM interpolated by EpicFlow is fast and robust for large displacements especially for fast moving objects and also preserves the edge structure well. Extensive experiments show that our algorithm is on par with the state-of-art optical flow methods on MPI-Sintel, KITTI and Middlebury.

Fast Optical Flow Estimation Without Parallel Architectures

According to recent results on Middlebury, MPI Sintel, and KITTI benchmarks, the accuracy of optical flow estimation algorithms has been significantly improved. The speed of them, however, has been too slow to meet the requirement of real-time applications. As a result, some parallel architectures (such as FPGA or GPU) have to be used for accelerating. Therefore, reducing the computational cost of optical flow estimation makes a lot of sense. To overcome the above issues, this paper proposes a fast local method based on 3D-gradients and approximate nearest-neighbor field (NNF), which is different from the widely used global model. In our method, NNF is used to provide initial optical flow field, and the proposed fast 3D-gradients-based local operator is used to propagate flow from coarse level to finer level in the coarse-to-fine refinement. We implement two versions of our method (with/without NNF initialization). Experimental results show that our method has a significant advantage for speed over other methods, where the fast version is capable of processing $Urban$ sequence ( $640\times 480$ ) at $\approx 10$ frames/s without parallel architectures. Meanwhile, our accuracy is also within acceptable levels on both small and large motion for some real-time applications.

Quality assurance of a gimbaled head swing verification using feature point tracking.

To perform dynamic tumor tracking (DTT) for clinical applications safely and accurately, gimbaled head swing verification is important. We propose a quantitative gimbaled head swing verification method for daily quality assurance (QA), which uses feature point tracking and a web camera. The web camera was placed on a couch at the same position for every gimbaled head swing verification, and could move based on a determined input function (sinusoidal patterns; amplitude: ± 20 mm; cycle: 3 s) in the pan and tilt directions at isocenter plane. Two continuous images were then analyzed for each feature point using the pyramidal Lucas–Kanade (LK) method, which is an optical flow estimation algorithm. We used a tapped hole as a feature point of the gimbaled head. The period and amplitude were analyzed to acquire a quantitative gimbaled head swing value for daily QA. The mean ± SD of the period were 3.00 ± 0.03 (range: 3.00–3.07) s and 3.00 ± 0.02 (range: 3.00–3.07) s in the pan and tilt directions, respectively. The mean ± SD of the relative displacement were 19.7 ± 0.08 (range: 19.6–19.8) mm and 18.9 ± 0.2 (range: 18.4–19.5) mm in the pan and tilt directions, respectively. The gimbaled head swing was reliable for DTT. We propose a quantitative gimbaled head swing verification method for daily QA using the feature point tracking method and a web camera. Our method can quantitatively assess the gimbaled head swing for daily QA from baseline values, measured at the time of acceptance and commissioning.

Cardiac Motion Estimation in Magnetic Resonance Images Using Optical Flow

This paper reports an optical based method for quantification of cardiac motion in MRI images. The cardiac motion is quantified in a Short-Axis (SAX) slice located at the mid-cavity of the left ventricle. In this slice, the left ventricle wall is segmented for extracting the endocardium, the epicardium and the mid‑wall. The velocity field for these contours is available after performing the optical flow estimation for the given slice. Post-processing of this motion field enables estimation of the radial displacement. The core of this cardiac motion quantification method is a sparse based algorithm for optical flow estimation. This algorithm is presented and compared with several classical optical flow estimation algorithms. The comparison is performed using several video sequences available from the Middlebury benchmark where the ground truth optical flow is known. The algorithms with better performance are also tested using a 4‑ D cardiac Magnetic Resonance Image (MRI) sequence. Results show that he sparse algorithm estimates an optical flow field that represents the motion of contraction during the systole interval. Moreover, results about radial displacement estimation on real MRI sequences for a normal subject and a patient with hypertrophy show that the proposed quantification method could be useful for quantification of left ventricle motion.

GCPOF: a novel optical flow estimation algorithm leveraged by sparse ground control points

As a fundamental task in computer vision, optical flow estimation algorithms aim to establish dense pixel correspondences between image frames. This paper presents a novel optical flow estimation framework called GCPOF to handle large displacement and scale variations of scene objects, which appear frequently and pose great challenges in practice. Within the framework of GCPOF, large displacement and scale variations are captured by a new problem formulation leveraged by sparse ground control points. We present detailed theoretical derivation of the solution to the problem based on iterative reweighted least squares. Both qualitative and quantitative evaluations on synthetic and real images demonstrate that GCPOF is able to handle optical flow fields with large displacement and scale variations properly, and it runs significantly faster than relevant optical flow estimation methods.

Motion estimation using the total variation-local-global optical flow and the structure-texture image decomposition

Motion estimation is currently approximated by the visual displacement field called optical flow. The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. Actually, several methods are used to estimate the optical flow, but a good compromise between computational cost and accuracy is hard to achieve. This work presents a combined local-global-total variation CLG-TV approach with structure-texture image decomposition. The combination is used to control the propagation phenomena and to gain robustness against illumination changes, influence of texture on the results and sensitivity to outliers. The resulting method is able to compute larger displacements in a reasonable time.

Optical flow estimation based on the structure–texture image decomposition

Optical flow approaches for motion estimation calculate vector fields which determine the apparent velocities of objects in time-varying image sequences. Image motion estimation is a fundamental issue in low-level vision and is used in many applications in image sequence processing, such as robot navigation, object tracking, image coding and structure reconstruction. The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. Actually, several methods are used to estimate the optical flow, but a good compromise between computational cost and accuracy is hard to achieve. This work presents a combined local–global total variation approach with structure–texture image decomposition. The combination is used to control the propagation phenomena and to gain robustness against illumination changes, influence of noise on the results and sensitivity to outliers. The resulted method is able to compute larger displacements in a reasonable time.

Blind motion deblurring using optical flow

In recent years, a lot of motion deblurring methods have been proposed, most of which divide the motion blur problem into two parts: camera shake and object motion, and just deal with only one side of the problem. As a result, in this paper we focus on seeking a effective model to handle these two kinds of motion blur simultaneously. An optical flow based model is then proposed. In particular, the blurred image can be formulated as an integration of some clear intermediate images after optical based transformed. Meanwhile an effective algorithm based on this new model is introduced to deblur different types of blurry images. In this work, we not only use the optical flow to represent the blur process but also derive the optical flow estimation algorithm to help estimate the non-uniform PSF for each pixel. In addition, we show that by incorporating different kinds of constraints our method can get more robust and stable results. The experiments on both synthesized and real images show that our algorithm can effectively recover the latent images and get comparable results with some state of the art algorithms.