Optical Flow Datasets Research Articles

Enhancing Infrared Optical Flow Network Computation through RGB-IR Cross-Modal Image Generation.

Due to the complexity of real optical flow capture, the existing research still has not performed real optical flow capture of infrared (IR) images with the production of an optical flow based on IR images, which makes the research and application of deep learning-based optical flow computation limited to the field of RGB images only. Therefore, in this paper, we propose a method to produce an optical flow dataset of IR images. We utilize the RGB-IR cross-modal image transformation network to rationally transform existing RGB image optical flow datasets. The RGB-IR cross-modal image transformation is based on the improved Pix2Pix implementation, and in the experiments, the network is validated and evaluated using the RGB-IR aligned bimodal dataset M3FD. Then, RGB-IR cross-modal transformation is performed on the existing RGB optical flow dataset KITTI, and the optical flow computation network is trained using the IR images generated by the transformation. Finally, the computational results of the optical flow computation network before and after training are analyzed based on the RGB-IR aligned bimodal data.

Self-supervised few-shot learning for real-time traffic sign classification

Although supervised approaches for traffic sign classification have demonstrated excellent performance, they are limited to classifying several traffic signs defined in the training dataset. This prevents them from being applied to different domains, i.e., different countries. Herein, we propose a self-supervised approach for few-shot learning-based traffic sign classification. A center-awareness similarity network is designed for the traffic sign problem and trained using an optical flow dataset. Unlike existing supervised traffic sign classification methods, the proposed method does not depend on traffic sign categories defined by the training dataset. It applies to any traffic signs from different countries. We construct a Korean traffic sign classification (KTSC) dataset, including 6000 traffic sign samples and 59 categories. We evaluate the proposed method with baseline methods using the KTSC, German traffic sign, and Belgian traffic sign classification datasets. Experimental results show that the proposed method extends the ability of existing supervised methods and can classify any traffic sign, regardless of region/country dependence. Furthermore, the proposed approach significantly outperforms baseline methods for patch similarity. This approach provides a flexible and robust solution for classifying traffic signs, allowing for accurate categorization of every traffic sign, regardless of regional or national differences.

PDC-Net+: Enhanced Probabilistic Dense Correspondence Network

Establishing robust and accurate correspondences between a pair of images is a long-standing computer vision problem with numerous applications. While classically dominated by sparse methods, emerging dense approaches offer a compelling alternative paradigm that avoids the keypoint detection step. However, dense flow estimation is often inaccurate in the case of large displacements, occlusions, or homogeneous regions. In order to apply dense methods to real-world applications, such as pose estimation, image manipulation, or 3D reconstruction, it is therefore crucial to estimate the confidence of the predicted matches. We propose the Enhanced Probabilistic Dense Correspondence Network, PDC-Net+, capable of estimating accurate dense correspondences along with a reliable confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and an enhanced training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the tasks of pose estimation, 3D reconstruction, image-based localization, and image retrieval. Code and models are available at https://github.com/PruneTruong/DenseMatching.

Physics-based optical flow estimation under varying illumination conditions

The physics-based optical flow (PBOF) model was derived from typical flow visualizations and it provided a more solid physical foundation for optical flow calculation. However, like many recent variational optical flow methods, it is not formulated to deal with the effect of illumination variation. In this paper, we propose a new efficient illumination-invariant optical flow estimation method, which improves the physics-based optical flow model. The proposed method introduces a multiplicative product of the velocity gradient and intensity from PBOF and an additive diffusion component to relax the brightness constancy assumption. The proposed method employs a linear transformation to reduce the influence of varying illumination on the data term. Then, a smoothness-sparsity regularization is used to constrain the minimization problem to have good edge-preserving for optical flow. Furthermore, the numerical implementation is given to solve the optimization problem. The accuracy of the developed method and the robustness against varying illumination are evaluated. The proposed method can provide motion estimation with acceptable accuracy for three optical flow datasets and cloud images of Jupiter’s great red spot, which have challenging illumination variations.

River Surface Velocimetry Based on Virtual River Dataset and Modulated-RAFT

Many image-based velocimetry methods, such as large-scale particle image velocimetry (LSPIV), space-time image velocimetry (STIV) and optical flow velocimetry (OFV), have been proposed to estimate river surface velocity (RSV) efficiently and precisely. Among these methods, OFV-based methods have drawn great attention due to its high field resolution and low requirement of tracers. As a powerful approach for estimate optical flow accurately and efficiently, deep optical flow estimation is utilized in OFV-based methods as well. However, these methods often use irrelevant datasets for training due to the immeasurability of optical flow. Models obtained by such approaches suffer from limited generalization because of domain drift. Besides, high similarity of river surfaces often leads to ambiguous correlation volume extracted by deep optical flow estimation model, which can cause mismatch. This work proposed a method for accurately and robustly RSV estimation under velocity range of 0-6.0 m/s. Specifically, we introduced a method for generating relative optical flow datasets and proposed MRAFT, a deep optical flow estimation model combining with correlation volume modulation. Experiment results demonstrate that the generated virtual river datasets effectively improve the generalization of the model and MRAFT remarkably alleviates the mismatch. Our work facilitates the application of deep optical flow on RSV estimation and provides other related researches with optical flow datasets for fine-tuning.

Optical Flow in the Dark

Optical flow estimation in low-light conditions is a challenging task for existing methods and current optical flow datasets lack low-light samples. Even if the dark images are enhanced before estimation, which could achieve great visual perception, it still leads to suboptimal optical flow results because information like motion consistency may be broken during the enhancement. We propose to apply a novel training policy to learn optical flow directly from new synthetic and real low-light images. Specifically, first, we design a method to collect a new optical flow dataset in multiple exposures with shared optical flow pseudo labels. Then we apply a two-step process to create a synthetic low-light optical flow dataset, based on an existing bright one, by simulating low-light raw features from the multi-exposure raw images we collected. To extend the data diversity, we also include published low-light raw videos without optical flow labels. In our training pipeline, with the three datasets, we create two teacher-student pairs to progressively obtain optical flow labels for all data. Finally, we apply a mix-up training policy with our diversified datasets to produce low-light-robust optical flow models for release. The experiments show that our method can relatively maintain the optical flow accuracy as the image exposure descends and the generalization ability of our method is tested with different cameras in multiple practical scenes.

Optical flow estimation of coronary angiography sequences based on semi-supervised learning

Optical flow is widely used in medical image processing, such as image registration, segmentation, 3D reconstruction, and temporal super-resolution. However, high-precision optical flow training datasets for medical images are challenging to produce. The current optical flow estimation models trained on these non-medical datasets, such as KITTI, Sintel, and FlyingChairs are unsuitable for medical images. In this work, we propose a semi-supervised learning mechanism to estimate the optical flow of coronary angiography. Our proposed method only needs the original medical images, segmentation results of regions of interest, and pre-trained models based on other optical flow datasets to train a new optical flow estimation model suitable for medical images. First, we use the coronary segmentation results to perform image enhancement processing on the coronary vascular region to improve the image contrast between the vascular region and the surrounding tissues. Then, we extract the high-precision optical flow of coronary arteries based on the coronary-enhanced images and the pre-trained optical flow estimation model. After estimating the optical flow, we take it and its corresponding original coronary angiography images as the training dataset to train the optical flow estimation network. Furthermore, we generate a large-scale synthetic Flying-artery dataset based on coronary artery segmentation results and original coronary angiography images, which is used to improve and evaluate the accuracy of optical flow estimation for coronary angiography. The experimental results on the coronary angiography datasets demonstrate that our proposed method can significantly improve the optical flow estimation accuracy of coronary angiography sequences compared with other methods.

Behind-The-Scenes (BTS): Wiper-Occlusion Canceling for Advanced Driver Assistance Systems in Adverse Rain Environments

Driving in an adverse rain environment is a crucial challenge for vision-based advanced driver assistance systems (ADAS) in the automotive industry. The vehicle windshield wiper removes adherent raindrops that cause distorted images from in-vehicle frontal view cameras, but, additionally, it causes an occlusion that can hinder visibility at the same time. The wiper-occlusion causes erroneous judgments by vision-based applications and endangers safety. This study proposes behind-the-scenes (BTS) that detects and removes wiper-occlusion in real-time image inputs under rainy weather conditions. The pixel-wise wiper masks are detected by high-pass filtering to predict the optical flow of a sequential image pair. We fine-tuned a deep learning-based optical flow model with a synthesized dataset, which was generated with pseudo-ground truth wiper masks and flows using auto-labeling with acquired real rainy images. A typical optical flow dataset with static synthetic objects is synthesized with real fast-moving objects to enhance data diversity. We annotated wiper masks and scenes as detection ground truths from the collected real images for evaluation. BTS outperforms by achieving a 0.962 SSIM and 91.6% F1 score in wiper mask detection and 88.3% F1 score in wiper image detection. Consequently, BTS enhanced the performance of vision-based image restoration and object detection applications by canceling occlusions and demonstrated it potential role in improving ADAS under rainy weather conditions.

Research Progress of Deep Optical Flow Estimation

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

Variational Densification and Refinement of Registration Maps

Local patch-based algorithms for image registration fail to accurately match points in areas not discriminative enough, mainly textureless regions. These methods normally involve a validation process and provide a non-completely dense solution. In this paper, we propose a novel refinement and completion approach for registration. The proposed model combines single image nonlocal densification with classical variational image registration. We associate a total variation regularization with a nonlocal term to provide a smooth solution leveraging the image geometry. We show experiments on public stereo and optical flow datasets to filter and densify incomplete depth maps and motion fields. Extensive comparisons against existing and state-of-the-art depth/motion fields densification approaches demonstrate the competitive performance of the introduced method. Additionally, we illustrate how our method can deal with other tasks, such as filtering and interpolation of depth maps from RGBD data and depth upsampling.

Self-Guided Stereo Correspondence Estimation Algorithm

This paper introduces an innovative algorithm, “Self-guided Stereo Correspondence” (SGSC), that is directed by photometric properties of the candidate pixels. As the photometric properties of reference image (left image) pixel and its neighbor’s pixel are similar in most cases, so the upcoming corresponding pixel exists in the surrounding of the previous matching pixel. Searching performance is greatly improved by utilizing this photometric property of the candidate pixels. The searching performance is further improved by applying the pioneering threshold technique. These two key techniques sufficiently reduced the computational cost with no degradation of accuracy. The achievements of the proposed method are testified on Middlebury standard Stereo Datasets of 2003 and 2006 and the Middlebury latest Optical Flow Dataset. Finally, the proposed method is compared with present state-of-the-art methods and our SGSC outdoes the latest methods in terms of speed, visualization of hidden ground truth, 3D reconstruction and accuracy.

Graph Laplacian Regularization for Robust Optical Flow Estimation.

This paper proposes graph Laplacian regularization for robust estimation of optical flow. First, we analyze the spectral properties of dense graph Laplacians and show that dense graphs achieve a better trade-off between preserving flow discontinuities and filtering noise, compared with the usual Laplacian. Using this analysis, we then propose a robust optical flow estimation method based on Gaussian graph Laplacians. We revisit the framework of iteratively reweighted least-squares from the perspective of graph edge reweighting, and employ the Welsch loss function to preserve flow discontinuities and handle occlusions. Our experiments using the Middlebury and MPI-Sintel optical flow datasets demonstrate the robustness and the efficiency of our proposed approach.

Low-dimensional superpixel descriptor and its application in visual correspondence estimation

Establishing local visual correspondence between video frames is an important and challenging problem in many vision based applications. Local keypoint detection and description based pixel-level matching is a typical way for visual correspondence estimation. Unlike traditional local keypoint descriptor based methods, this paper proposes a comprehensive yet low-dimensional local feature descriptor based on superpixels generated by over segmentation. The proposed local feature descriptor extracts shape feature, texture feature, and color feature from superpixels by orientated center-boundary distance (OCBD), gray-level co-occurrence matrix (GLCM), and saturation histogram (SHIST), respectively. The types of features are more comprehensive than existing descriptors which extract only one specific kind of feature. Experimental results on the widely used Middlebury optical flow dataset prove that the proposed superpixel descriptor achieves triple accuracy compared with the state-of-the-art ORB descriptor which has the same dimension of features with the proposed one. In addition, since the dimension of the proposed superpixel descriptor is low, it is convenient for matching and memory-efficient for hardware implementation.

Illumination-Invariance Optical Flow Estimation Using Weighted Regularization Transform

Many recent variational optical flow methods are not robust for illumination variance, and they only consider local image relation in terms of illumination. In this paper, we propose a new efficient illumination-invariance total variation optical flow method called the weighted regularization transform, which uses and optimizes the Weber’s Law. Our method exploits unequal probability as the weight that has non-local information to estimate stable optical flow despite illumination changes. The proposed method uses a coarse-to-fine pyramid model to reduce the influence on the data term from illumination. Then, an energy optimization procedure is introduced to constrain the minimization of the data term with the non-local regularization. Experimentation with the proposed method has been performed on three optical flow datasets and a face liveness detection database, which have challenging illumination variations, and the results demonstrate that the proposed method is quite robust with respect to variations in illumination.

Rapid handling of outliers in dense sampling descriptor correspondence fields

The sparse-to-dense approach is considered to be the standard method of capturing the long-range motion of small objects during large displacement optical flow. Despite progress in the matching and interpolation of this approach, little work has focused on improving the handling of outliers after dense sampling descriptor matching. We propose an improved grid-based statistical matching method that can quickly remove outliers without calculating backward flow. First, a multigrid statistical matching method is developed to remove the most outliers of the dense sampling descriptor correspondence field. Second, to improve the accuracy of outliers handling, the misjudgment match in the edge grid is corrected based on the statistical matching constraint. The results of extensive experiments on public optical flow datasets demonstrate the effectiveness of the proposed method.

A Dataset for Visual Navigation with Neuromorphic Methods.

Standardized benchmarks in Computer Vision have greatly contributed to the advance of approaches to many problems in the field. If we want to enhance the visibility of event-driven vision and increase its impact, we will need benchmarks that allow comparison among different neuromorphic methods as well as comparison to Computer Vision conventional approaches. We present datasets to evaluate the accuracy of frame-free and frame-based approaches for tasks of visual navigation. Similar to conventional Computer Vision datasets, we provide synthetic and real scenes, with the synthetic data created with graphics packages, and the real data recorded using a mobile robotic platform carrying a dynamic and active pixel vision sensor (DAVIS) and an RGB+Depth sensor. For both datasets the cameras move with a rigid motion in a static scene, and the data includes the images, events, optic flow, 3D camera motion, and the depth of the scene, along with calibration procedures. Finally, we also provide simulated event data generated synthetically from well-known frame-based optical flow datasets.

Nonlinear Transform for Robust Dense Block-Based Motion Estimation

We present a noniterative multiresolution motion estimation strategy, involving block-based comparisons in each detail band of a Laplacian pyramid. A novel matching score is developed and analyzed. The proposed matching score is based on a class of nonlinear transformations of Laplacian detail bands, yielding 1-bit or 2-bit representations. The matching score is evaluated in a dense full-search motion estimation setting, with synthetic video frames and an optical flow data set. Together with a strategy for combining the matching scores across resolutions, the proposed method is shown to produce smoother and more robust estimates than mean square error (MSE) in each detail band and combined. It tolerates more of nontranslational motion, such as rotation, validating the analysis, while providing much better localization of the motion discontinuities. We also provide an efficient implementation of the motion estimation strategy and show that the computational complexity of the approach is closely related to the traditional MSE block-based full-search motion estimation procedure.

Efficient Hardware Implementation of the Horn-Schunck Algorithm for High-Resolution Real-Time Dense Optical Flow Sensor

This article presents an efficient hardware implementation of the Horn-Schunck algorithm that can be used in an embedded optical flow sensor. An architecture is proposed, that realises the iterative Horn-Schunck algorithm in a pipelined manner. This modification allows to achieve data throughput of 175 MPixels/s and makes processing of Full HD video stream (1, 920 × 1, 080 @ 60 fps) possible. The structure of the optical flow module as well as pre- and post-filtering blocks and a flow reliability computation unit is described in details. Three versions of optical flow modules, with different numerical precision, working frequency and obtained results accuracy are proposed. The errors caused by switching from floating- to fixed-point computations are also evaluated. The described architecture was tested on popular sequences from an optical flow dataset of the Middlebury University. It achieves state-of-the-art results among hardware implementations of single scale methods. The designed fixed-point architecture achieves performance of 418 GOPS with power efficiency of 34 GOPS/W. The proposed floating-point module achieves 103 GFLOPS, with power efficiency of 24 GFLOPS/W. Moreover, a 100 times speedup compared to a modern CPU with SIMD support is reported. A complete, working vision system realized on Xilinx VC707 evaluation board is also presented. It is able to compute optical flow for Full HD video stream received from an HDMI camera in real-time. The obtained results prove that FPGA devices are an ideal platform for embedded vision systems.

Constrained Optical Flow Estimation as a Matching Problem

In general, discretization in the motion vector domain yields an intractable number of labels. In this paper, we propose an approach that can reduce general optical flow to the constrained matching problem by pre-estimating a 2-D disparity labeling map of the desired discrete motion vector function. One of the goals of the proposed paper is estimating coarse distribution of motion vectors and then utilizing this distribution as global constraints for discrete optical flow estimation. This pre-estimation is done with a simple frame-to-frame correlation technique also known as the digital symmetric-phase-only-filter (SPOF). We discover a strong correlation between the output of the SPOF and the motion vector distribution of the related optical flow. A two step matching paradigm for optical flow estimation is applied: pixel accuracy (integer flow) and subpixel accuracy estimation. The matching problem is solved by global optimization. Experiments on the Middlebury optical flow datasets confirm our intuitive assumptions about strong correlation between motion vector distribution of optical flow and maximal peaks of SPOF outputs. The overall performance of the proposed method is promising and achieves state-of-the-art results on the Middlebury benchmark.

Prediction of feather damage in laying hens using optical flows and Markov models

Feather pecking in laying hens is a major welfare and production problem for commercial egg producers, resulting in mortality, loss of production as well as welfare issues for the damaged birds. Damaging outbreaks of feather pecking are currently impossible to control, despite a number of proposed interventions. However, the ability to predict feather damage in advance would be a valuable research tool for identifying which management or environmental factors could be the most effective interventions at different ages. This paper proposes a framework for forecasting the damage caused by injurious pecking based on automated image processing and statistical analysis. By frame-by-frame analysis of video recordings of laying hen flocks, optical flow measures are calculated as indicators of the movement of the birds. From the optical flow datasets, measures of disturbance are extracted using hidden Markov models. Based on these disturbance measures and age-related variables, the levels of feather damage in flocks in future weeks is predicted. Applying the proposed method to real-world datasets, it is shown that the disturbance measures offer improved predictive values for feather damage thus enabling an identification of flocks with probable prevalence of damage and injury later in lay.