Optical Flow Estimation Research Articles

SA‐FlowNet: Event‐based self‐attention optical flow estimation with spiking‐analogue neural networks

AbstractInspired by biological vision mechanism, event‐based cameras have been developed to capture continuous object motion and detect brightness changes independently and asynchronously, which overcome the limitations of traditional frame‐based cameras. Complementarily, spiking neural networks (SNNs) offer asynchronous computations and exploit the inherent sparseness of spatio‐temporal events. Notably, event‐based pixel‐wise optical flow estimations calculate the positions and relationships of objects in adjacent frames; however, as event camera outputs are sparse and uneven, dense scene information is difficult to generate and the local receptive fields of the neural network also lead to poor moving objects tracking. To address these issues, an improved event‐based self‐attention optical flow estimation network (SA‐FlowNet) that independently uses criss‐cross and temporal self‐attention mechanisms, directly capturing long‐range dependencies and efficiently extracting the temporal and spatial features from the event streams is proposed. In the former mechanism, a cross‐domain attention scheme dynamically fusing the temporal‐spatial features is introduced. The proposed network adopts a spiking‐analogue neural network architecture using an end‐to‐end learning method and gains significant computational energy benefits especially for SNNs. The state‐of‐the‐art results of the error rate for optical flow prediction on the Multi‐Vehicle Stereo Event Camera (MVSEC) dataset compared with the current SNN‐based approaches is demonstrated.

Optical flow estimation from event-based cameras and spiking neural networks.

Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or "spikes", when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency, and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates.

Optical Flow and Expansion Based Deep Temporal Up-Sampling of LIDAR Point Clouds

This paper proposes a framework that enables the online generation of virtual point clouds relying only on previous camera and point clouds and current camera measurements. The continuous usage of the pipeline generating virtual LIDAR measurements makes the temporal up-sampling of point clouds possible. The only requirement of the system is a camera with a higher frame rate than the LIDAR equipped to the same vehicle, which is usually provided. The pipeline first utilizes optical flow estimations from the available camera frames. Next, optical expansion is used to upgrade it to 3D scene flow. Following that, ground plane fitting is made on the previous LIDAR point cloud. Finally, the estimated scene flow is applied to the previously measured object points to generate the new point cloud. The framework’s efficiency is proved as state-of-the-art performance is achieved on the KITTI dataset.

Feature Correlation Transformer for Estimating Ambiguous Optical Flow

Cost volume is widely used to establish correspondences in optical flow estimation. However, when dealing with low-texture and occluded areas, it is difficult to estimate the cost volume correctly. Therefore, we propose a replacement: feature correlation transformer (FCTR), a transformer with self- and cross-attention alternations for obtaining global receptive fields and positional embedding for establishing correspondences. With global context and positional information, FCTR can produce more accurate correspondences for ambiguous areas. Using position-embedded feature allows the removal of the context network; the positional information can be aggregated within ambiguous motion boundaries, and the number of model parameters can be reduced. To speed up network convergence and strengthen robustness, we introduce a smooth L1 loss with exponential weights in the pre-training step. At the time of submission, our method achieves competitive performance with all published optical flow methods on both the KITTI-2015 and MPI-Sintel benchmarks. Moreover, it outperforms all optical flow and scene flow methods in KITTI-2015 foreground-region prediction.

Self-adaptive optical flow estimation in multiharsh outdoor scenes

Open-air optical flow estimation is crucial for autonomous driving and traffic monitoring. However, it is still challenging to obtain high-precision optical flow under severe imaging environments. To alleviate this problem, we propose a self-adaptive optical flow estimation framework for harsh outdoor scenes, which consists of a frame-adaptive learnable module (FAL-module), a streak-invariant feature extractor (SFE), and a feature matching and flow propagation network. In the framework, three components are jointly learned in an end-to-end manner. The input frames can be adaptively enhanced without sacrificing optical flow-related information by FAL-module. And the SFE decreases the influence of streak-like noise generated by rainy and snowy. Compared with the state-of-the-art methods, the experimental results show that our approach can significantly improve the optical flow estimation quality in multiple harsh outdoor scenes without performance penalties in normal scenes.

Regularization for Unsupervised Learning of Optical Flow.

Regularization is an important technique for training deep neural networks. In this paper, we propose a novel shared-weight teacher-student strategy and a content-aware regularization (CAR) module. Based on a tiny, learnable, content-aware mask, CAR is randomly applied to some channels in the convolutional layers during training to be able to guide predictions in a shared-weight teacher-student strategy. CAR prevents motion estimation methods in unsupervised learning from co-adaptation. Extensive experiments on optical flow and scene flow estimation show that our method significantly improves on the performance of the original networks and surpasses other popular regularization methods. The method also surpasses all variants with similar architectures and the supervised PWC-Net on MPI-Sintel and on KITTI. Our method shows strong cross-dataset generalization, i.e., our method solely trained on MPI-Sintel outperforms a similarly trained supervised PWC-Net by 27.9% and 32.9% on KITTI, respectively. Our method uses fewer parameters and less computation, and has faster inference times than the original PWC-Net.

A Novel Moving Object Detection Algorithm Based on Robust Image Feature Threshold Segmentation with Improved Optical Flow Estimation

The detection of moving objects in images is a crucial research objective; however, several challenges, such as low accuracy, background fixing or moving, ‘ghost’ issues, and warping, exist in its execution. The majority of approaches operate with a fixed camera. This study proposes a robust feature threshold moving object identification and segmentation method with enhanced optical flow estimation to overcome these challenges. Unlike most optical flow Otsu segmentation for fixed cameras, a background feature threshold segmentation technique based on a combination of the Horn–Schunck (HS) and Lucas–Kanade (LK) optical flow methods is presented in this paper. This approach aims to obtain the segmentation of moving objects. First, the HS and LK optical flows with the image pyramid are integrated to establish the high-precision and anti-interference optical flow estimation equation. Next, the Delaunay triangulation is used to solve the motion occlusion problem. Finally, the proposed robust feature threshold segmentation method is applied to the optical flow field to attract the moving object, which is the. extracted from the Harris feature and the image background affine transformation model. The technique uses morphological image processing to create the final moving target foreground area. Experimental results verified that this method successfully detected and segmented objects with high accuracy when the camera was either fixed or moving.

Deep Learning Optical Flow with Compound Loss for Dense Fluid Motion Estimation

Flow motion with complex patterns, such as vortex, stagnant flow, and seepage, put forward higher spatial resolution requirements for particle image velocimetry (PIV). With the development of deep learning technology in optical flow estimation, many attempts have been made to introduce deep learning-based optical flow (DLOF) into PIV. Compared with the traditional optical flow method, the DLOF method has the advantages of higher precision, faster calculation speed, and avoiding manual parameter adjustment. However, DLOF research is generally developed based on the basic characteristics of rigid body motion, and its key loss function part still generally uses the L1 (mean absolute error, MAE, L1) or L2 (mean square error, MSE, L2) loss functions, which lack consideration of fluid motion characteristics. Therefore, the current DLOF research has the problems of large angular error and serious curl-divergence loss in fluid motion estimation scenarios with smaller spatial scales than rigid bodies. Based on the prior knowledge of the traditional fluid motion characteristics, this study proposes a fluid loss function for describing the fluid motion characteristics, and combines this loss function with Flownet2. The compound loss (CL) function is combined with the displacement error, angular error, and div-curl smooth loss. The method combined with the loss function in this paper is called FlowNet2-CL-PIV. In order to verify that the compound loss function proposed in this study has a positive impact on the model training results, this paper uses the cosine similarity measure to demonstrate its effectiveness. In addition, the parameter selection of the compound loss function is analyzed and compared, and it is verified that the best training effect can be achieved by adjusting the parameter so that the order of magnitude of each part of the compound loss function is consistent. In order to test the calculation effect of the Flownet2-CL-PIV method proposed in this study, synthetic particle images are used for model training and performance analysis. Simulation results in various flow fields show that the root mean square error (RMSE) and average angular error (AAE) of Flownet2-CL-PIV reach 0.182 pixels and 1.7°, which are 10% and 54% higher than the original model, respectively.

MOP-SLAM: A real time SLAM system based on multi-head optical flow estimation network

SLAM (simultaneous localization and mapping) is becoming a significant technology in driverless vehicles nowadays. Sometimes the optical flow method can be used to track feature points to reduce the amount of calculation. But the traditional optical flow method is based on the assumption that the luminosity will not change and has high requirements for the environment. Therefore, an optical flow calculation method based on neural network called Optical-Net is proposed in this paper to match and track feature and corner points. We use the multi-head attention module to extract and fuse the features of different scales. The training results show that the Optical-Net designed in this study effectively improves the robustness and accuracy of optical flow estimation when the object has large displacement and small displacement. Meanwhile, to address the problem of reduced matching accuracy during video input due to dynamic blurring, a video dynamic deblurring algorithm based on the one-dimensional Wineman filter is adopted to preprocess the input video. Finally, we replaced the tracking thread in traditional SLAM. The performance of the improved algorithm is verified by experiments on our own driverless car.

Estimating Optical Flow with Streaming Perception and Changing Trend Aiming to Complex Scenarios

Most of the recent works on optical flow estimation focus on introducing more effective CNNs or even the Transformer architecture that is quite popular in today’s CV field, but they all still follow the traditional two-frame input structure. In this paper, we argue that optical flow estimation should be more focused on the dynamic information in the image sequence. To that end, we present a straightforward but effective streaming framework equipped with a novel prediction module for capturing optical flow trends, which is a point that has not yet been discussed in previous works. By employing the backbone of the strong baseline to extract the stream of features from three consecutive frames as inputs, this module can predict the current state and update the previous changing trend of optical flow for feature fusion at the next stage. Additionally, we redesign the training loss, providing two new cost functions, in particular, to measure the variations in the optical flow trend. At the forward stage, we deploy a historical buffer to maintain the inference speed so that it is unaffected by additional input frames. Compared with the strong baseline, our method improves the average end-point error by 21.12% on Sintel Clean and 12.94% on Sintel Final. In addition, it also outperforms existing methods in real-world scenarios with low brightness or fast-moving targets.

Decoupled spatiotemporal adaptive fusion network for self-supervised motion estimation

Optical flow estimation searches for correspondence between two images. In the unsupervised approach, most networks use the feature correlation volume to track the flow, and unsupervised training is achieved through a photometric loss function. However, various complex situations in the natural environment, such as object occlusion, motion blur, the camera being out-of-focus, limited perspective, and variation in lighting conditions, make it challenging to find correspondence accurately, thus complicating unsupervised optical flow estimation. This study decouples the problem into two sub-tasks: one is to search for determined correspondence within a pair of frames, and the other is to cope with mismatched regions due to occlusion, blur, light variation, etc., by introducing more spatial and temporal context information. We propose a multi-frame temporal dynamic model that recursively infers optical flow over causal sequences of arbitrary-length. Our innovative approach introduces information entropy and forward–backward consistency checks to measure the confidence regarding the matching of image pairs. To compensate for low-confidence regions, the proposed network adaptively identifies regions with correspondence confidence and utilizes temporal and spatial smoothness assumptions for motion re-prediction. Paired with well-designed simulation of dynamic occlusion pseudo-labels and scene variation, our model can learn a variety of complex scenes in a multi-frame environment to optimize low-confidence regions efficiently. Experimental results demonstrate that the proposed model is able to run at high speed in real-time tasks while maintaining high accuracy, thus achieving state-of-the-art results on Sintel Clean and Final benchmarks.

Uncertainty-Driven Dense Two-View Structure From Motion

This work introducesan effective and practical solution to the dense two-view structure from motion (S <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</i> M) problem. One vital question addressed is how to mindfully use per-pixel optical flow correspondence betweentwo frames for accurate pose estimation—as perfect per-pixel correspondence between two images is difficult, if not impossible, to establish. With the carefully estimated camera pose and predicted per-pixel optical flow correspondences, a dense depth of the scene is computed. Later, an iterative refinement procedure is introduced to further improve optical flow matching confidence, camera pose, and depth, exploiting their inherent dependency in rigid S <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</i> M. The fundamental idea presented is to benefit from per-pixel uncertainty in the optical flow estimation and provide robustness to the dense S <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</i> M system via an online refinement. Concretely, we introduce a pipeline consisting of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(i)</i> an uncertainty-aware dense optical flow estimation approach that provides per-pixel correspondence with their confidence score of matching; <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(ii)</i> a weighted dense bundle adjustment formulation that depends on optical flow uncertainty and bidirectional optical flow consistency to refine both pose and depth; <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(iii)</i> A depth estimation network that considers its consistency with the estimated poses and optical flow respecting epipolar constraint. Extensive experiments show that the proposed approach achieves remarkable depth accuracy and state-of-the-art camera pose results superseding SuperPoint and SuperGlue accuracy when tested on benchmark datasets such as DeMoN, YFCC100 M, and ScanNet.

Automated crack detection in laminated composites by optical flow measurements

In this paper, an automated crack detection method for laminated composites of carbon fiber reinforced polymers (CFRP) is presented. Image correlation is performed on micrographs of the unspeckled specimen edge using DeepFlow, an algorithm for optical flow estimation. Cracks are then detected from local maxima of the displacement gradient evaluated along the midline of the plies. The detection strategy is applied to 3 different materials, with increasingly irregular microstructures, and on quasi-isotropic layups (a challenging case for crack detection). The crack density evolution in the 90∘ and ±45∘ plies is presented. The proposed method proves its capacity to detect cracks, even in complex configurations where other methods fail, and can provide an extensive amount of data that are useful for the identification of damage models for laminates. The procedure is largely automated in order to make it suitable for industrial applications.

Low-cost real-time VLSI system for high-accuracy optical flow estimation using biological motion features and random forests

Low-cost real-time VLSI system for high-accuracy optical flow estimation using biological motion features and random forests

RivQNet: Deep Learning Based River Discharge Estimation Using Close‐Range Water Surface Imagery

AbstractStreamflow data is often the most critical input for hydrologic and hydraulic research, modeling, and design studies. Streamflow measurement using close range non‐contact sensing such as image velocimetry is a new technique that is yet far from maturity. Most current image‐based surface velocimetry techniques use correlation approaches that require user input to run the algorithms. This input can bias results if the operator is inexperienced. The main goal of this study is to develop a novel, accurate and fast river velocimetry scheme called RivQNet that does not require subjective user input. RivQNet processes close‐range non‐contact water surface images using artificial intelligence techniques. The algorithm is a deep‐learning optical flow estimation using a preferred available convolutional neural network architecture (i.e., FlowNet architecture). In this study the presented method is validated with common standard measurement methods and compared with conventional optical flow methodologies. The results indicate that the presented method yields accurate and dense spatial distributions of surface velocities.

LCIF-Net: Local criss-cross attention based optical flow method using multi-scale image features and feature pyramid

Although CNN-based optical flow methods have achieved remarkable performance in terms of computational accuracy and efficiency, the issue of edge-blurring caused by large displacements remains an open challenge. To address this problem, we propose a local criss-cross attention based optical flow estimation method using multi-scale image features and feature pyramid. First, we design an image pyramid-based feature extraction sub-network and then incorporate it into the feature pyramid network to construct a hybrid feature extraction module, which is able to extract multi-scale structural and semantic information from the input images. Second, we concatenate a local criss-cross attention module with the hybrid feature extraction module to build a global feature encoder. The global feature encoder further captures the long-range dependencies within the feature map to improve the large displacement estimation performance. Finally, we combine the global feature encoder with an iterative optical flow decoder, and thus propose a novel network named LCIF-Net. We demonstrate its significant performance benefits on MPI-Sintel and KITTI datasets. Compared with the existing optical flow estimation methods, our LCIF-Net remarkably improves the accuracy and robustness for the optical flow estimation, especially in the regions with large displacements and motion edges.

Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation

Video prediction is the problem of generating future frames by exploiting the spatiotemporal correlation from the past frame sequence. It is one of the crucial issues in computer vision and has many real-world applications, mainly focused on predicting future scenarios to avoid undesirable outcomes. However, modeling future image content and object is challenging due to the dynamic evolution and complexity of the scene, such as occlusions, camera movements, delay and illumination. Direct frame synthesis or optical-flow estimation are common approaches used by researchers. However, researchers mainly focused on video prediction using one of the approaches. Both methods have limitations, such as direct frame synthesis, usually face blurry prediction due to complex pixel distributions in the scene, and optical-flow estimation, usually produce artifacts due to large object displacements or obstructions in the clip. In this paper, we constructed a deep neural network Frame Prediction Network (FPNet-OF) with multiple-branch inputs (optical flow and original frame) to predict the future video frame by adaptively fusing the future object-motion with the future frame generator. The key idea is to jointly optimize direct RGB frame synthesis and dense optical flow estimation to generate a superior video prediction network. Using various real-world datasets, we experimentally verify that our proposed framework can produce high-level video frame compared to other state-of-the-art framework.

Self-Sufficient Feature Enhancing Networks for Video Salient Object Detection

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

Test-Time Adaptation for Optical Flow Estimation Using Motion Vectors.

Due to the prohibitive cost as well as technical challenges in annotating ground-truth optical flow for large-scale realistic video datasets, the existing deep learning models for optical flow estimation mostly rely on synthetic data for training, which in turn may lead to significant performance degradation under test-data distribution shift in real-world environments. In this work, we propose the methodology to tackle this important problem. We design a self-supervised learning task for adjusting the optical flow estimation model at test time. We exploit the fact that most videos are stored in compressed formats, from which compact information on motion, in the form of motion vectors and residuals, can be made readily available. We formulate the self-supervised task as motion vector prediction, and link this task to optical flow estimation. To the best of our knowledge, our Test-Time Adaption guided with Motion Vectors (TTA-MV), is the first work to perform such adaptation for optical flow. The experimental results demonstrate that TTA-MV can improve the generalization capability of various well-known deep learning methods for optical flow estimation, such as FlowNet, PWCNet, and RAFT.

Optical Flow Computation for Video Under the Dynamic Illumination

Optical flow computation for video under the dynamic illumination is a challenging issue in video multimedia applications. In this paper, we solve this issue by introducing an illumination-invariant framework for variational optical flow estimation. It consists of an illumination-invariance model that handles complex illumination changes and a data enhancement model that guarantees highly accurate optical flow estimation. In this framework, we design a log-correlation descriptor for the data term, which handles complex illumination changes by eliminating the common parameters shared by the neighboring pixels in the corresponding illumination model while improving the accuracy of optical flow estimation by enhancing the discriminability of the data term matching. We also introduce a novel optical flow model with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{0}$</tex-math></inline-formula> regularizer, which reconstructs optical flow field by a sparse flow gradient counting scheme. Different from other edge-preserving regularizers, it does not depend on local motion features, but locates important flow edges globally. Therefore, it will not cause edge blurriness due to avoiding local filtering or average operation. It is particularly effective for enhancing major flow edges while eliminating a manageable degree of low-amplitude motion structures to control smoothing and reduce oversegmentation artifacts. Even small-scale motion structures with high contrast can be preserved remarkably well. The experimental results show our method significantly outperforms previous illumination-robust optical flow methods in handling complex illumination changes, and achieves competitive evaluation results on the challenging MPI-Sintel and Kitti datasets.