Optical Flow Algorithm Research Articles

Initial evaluation of 2D and 3D simulated high-speed 1000 fps vascular contrast-flow image sequences using computational fluid dynamics (CFD).

Digital subtraction angiography (DSA) remains the clinical standard for detailed visualization of the neurovasculature due to its high-spatial resolution; however, detailed blood-flow quantification is impaired by its low-temporal resolution. Advances in photon-counting detector technology have led us to develop High-Speed Angiography (HSA), where x-ray images are acquired at 1000 fps for more accurate visualization and quantification of blood flow. We have implemented a physics-based optical flow method to extract such information from HSA, but validation of the angiography-derived velocity distributions is not straightforward. Computational fluid dynamics (CFD) is widely regarded as the benchmark for hemodynamic analysis, as it provides a multitude of quantitative flow parameters throughout the volume of interest. However, there are several limitations with this method related to over-simplification of boundary conditions and suboptimal meshing (spatial resolution), that make CFD simulation results an inexact criterion for validation. To overcome this issue for HSA validation, CFD was used to generate both simulated high-speed angiograms and the corresponding ground-truth 3D flow fields to better understand the relationship between the 3D volumetric-flow distribution and the 2D projected-flow distribution as is obtained with angiography, and the subsequent 2D approximation of flow velocity. Several geometries were investigated, ranging from simple pipe models to complex patient-specific aneurysms. Simulated datasets were analyzed with the optical flow algorithm, and the effects of flow divergence, quantum mottle, and intensity gradient on the calculation were evaluated. From these simulations, we can evaluate whether flow fields reconstructed from HSA are representative of significant flow patterns in the 3D vasculature.

Marker‐free vision‐based method for vibration measurements of RC structure under seismic vibration

AbstractThe seismic vibration measurement of reinforced concrete (RC) structures is an important aspect of post‐disaster reconstruction. Unfortunately, the existing method of installing sensors on walls and columns is prohibitively expensive, and the measurement signals are prone to loss during seismic vibration. In this paper, a marker‐free measurement method based on the Lucas‐Kanade (L‐K) optical flow algorithm for vibration measurements of RC structures under seismic vibration has been developed. In addition, to improve the accuracy of L‐K optical flow in dealing with the RC structure under seismic vibration, a new approach for selecting the feature point that is used to solve the restrictions of manual marking, as well as an adaptive matched window strategy that uses different sizes of matched window along the motion to replace the fixed matched window in the original L‐K algorithm, are proposed. The precision of the proposed method is first validated by comparing it to the original L‐K optical flow algorithm. Then, compare its precision and efficiency to the existing vision‐based methods, including feature matching (FM) and digital image correlation (DIC). Finally, the behaviors of RC structures during seismic vibration, such as inter‐story drift, frequency, and strain, are investigated. The results show that, when compared to the original L‐K optical flow algorithm and other vision‐based methods, the proposed method can accurately and efficiently measure motion without spraying speckle or high contrast markers on the structure's surface, and it enables the rapid diagnosis of structures affected by earthquakes.

Two-Dimensional Flow Field Measurement Method for Sediment-Laden Flow Based on Optical Flow Algorithm

This paper proposes a novel particle image velocimetry (PIV) technique to generate an instantaneous two-dimensional velocity field for sediment-laden fluid based on the optical flow algorithm of ultrasound imaging. In this paper, an ultrasonic PIV (UIV) system is constructed by integrating a medical ultrasound instrument and an ultrasonic particle image velocimetry algorithm. The medical ultrasound instrument with a phased sensor array is used to acquire acoustic echo signals of particles in water and generate two-dimensional underwater ultrasound images. Based on the optical flow field, the instantaneous velocity of the particles in water corresponding to the pixels in the ultrasonic particle images is derived from the grayscale change between adjacent images under the L-K local constraint based on the optical flow field, and finally, the two-dimensional flow field is obtained. Through multiple sets of experiments, the proposed algorithm is verified. The experimental results are compared with those of the conventional cross-correlation algorithms. The results show that the L-K optical flow method can not only obtain the underwater velocity field accurately, but also has the advantages of good smoothness and extensive suitability, especially for the flow field measurement in sediment-laden fluid than conventional algorithms.

RPEOD: A Real-Time Pose Estimation and Object Detection System for Aerial Robot Target Tracking

Pose estimation and environmental perception are the fundamental capabilities of autonomous robots. In this paper, a novel real-time pose estimation and object detection (RPEOD) strategy for aerial robot target tracking is presented. The aerial robot is equipped with a binocular fisheye camera for pose estimation and a depth camera to capture the spatial position of the tracked target. The RPEOD system uses a sparse optical flow algorithm to track image corner features, and the local bundle adjustment is restricted in a sliding window. Ulteriorly, we proposed YZNet, a lightweight neural inference structure, and took it as the backbone in YOLOV5 (the state-of-the-art real-time object detector). The RPEOD system can dramatically reduce the computational complexity in reprojection error minimization and the neural network inference process; Thus, it can calculate real-time on the onboard computer carried by the aerial robot. The RPEOD system is evaluated using both simulated and real-world experiments, demonstrating clear advantages over state-of-the-art approaches, and is significantly more fast.

Cells on Hydrogels with Micron-Scaled Stiffness Patterns Demonstrate Local Stiffness Sensing.

Cell rigidity sensing—a basic cellular process allowing cells to adapt to mechanical cues—involves cell capabilities exerting force on the extracellular environment. In vivo, cells are exposed to multi-scaled heterogeneities in the mechanical properties of the surroundings. Here, we investigate whether cells are able to sense micron-scaled stiffness textures by measuring the forces they transmit to the extracellular matrix. To this end, we propose an efficient photochemistry of polyacrylamide hydrogels to design micron-scale stiffness patterns with kPa/µm gradients. Additionally, we propose an original protocol for the surface coating of adhesion proteins, which allows tuning the surface density from fully coupled to fully independent of the stiffness pattern. This evidences that cells pull on their surroundings by adjusting the level of stress to the micron-scaled stiffness. This conclusion was achieved through improvements in the traction force microscopy technique, e.g., adapting to substrates with a non-uniform stiffness and achieving a submicron resolution thanks to the implementation of a pyramidal optical flow algorithm. These developments provide tools for enhancing the current understanding of the contribution of stiffness alterations in many pathologies, including cancer.

A comparison of satellite cloud motion vectors techniques to forecast intra-day hourly solar global horizontal irradiation

Solar forecasting provides valuable information for grid management. Satellite-based forecasting tools account for the short-term intra-day time horizons, typically outperforming numerical weather predictions up to 4–5 h ahead. The method consist of three separated stages, namely, cloud motion estimation, motion extrapolation and satellite-to-irradiation conversion. In this work we compare different satellite-based proposals for hourly irradiation forecast up to 5 h ahead using a 2-years data set. The widely-used Lorenz’s block matching technique and four optical flow (OF) algorithms are assessed, both at image and irradiation levels. All the methods are locally optimized to obtain their peak performance. It is found that the OF algorithm which combines an L1 data penalty term on the optical flow equation with total variation regularization (TVL1) outperforms the rest. Different image extrapolation approaches and spatial smoothing are also tested. It is found that changing the extrapolation technique does not have much impact in the overall performance and that important gains can be obtained by optimally smoothing the predicted images previous to solar irradiation conversion. By doing this, all methods outperform the exigent convex persistence benchmark, achieving positive forecasting skills. The tests are performed using GOES-East satellite images of south-east South America, and the methods’ optimal parameters are given.

Robust optical flow algorithm for general single cell segmentation.

Cell segmentation is crucial to the field of cell biology, as the accurate extraction of single-cell morphology, migration, and ultimately behavior from time-lapse live cell imagery are of paramount importance to elucidate and understand basic cellular processes. In an effort to increase available segmentation tools that can perform across research groups and platforms, we introduce a novel segmentation approach centered around optical flow and show that it achieves robust segmentation of single cells by validating it on multiple cell types, phenotypes, optical modalities, and in-vitro environments with or without labels. By leveraging cell movement in time-lapse imagery as a means to distinguish cells from their background and augmenting the output with machine vision operations, our algorithm reduces the number of adjustable parameters needed for manual optimization to two. We show that this approach offers the advantage of quicker processing times compared to contemporary machine learning based methods that require manual labeling for training, and in most cases achieves higher quality segmentation as well. This algorithm is packaged within MATLAB, offering an accessible means for general cell segmentation in a time-efficient manner.

Performance Evaluation of Deformable Image Registration Algorithms Using Computed Tomography of Multiple Lung Metastases.

Purpose: Various deformable image registration (DIR) methods have been used to evaluate organ deformations in 4-dimensional computed tomography (4D CT) images scanned during the respiratory motions of a patient. This study assesses the performance of 10 DIR algorithms using 4D CT images of 5 patients with fiducial markers (FMs) implanted during the postoperative radiosurgery of multiple lung metastases. Methods: To evaluate DIR algorithms, 4D CT images of 5 patients were used, and ground-truths of FMs and tumors were generated by physicians based on their medical expertise. The positions of FMs and tumors in each 4D CT phase image were determined using 10 DIR algorithms, and the deformed results were compared with ground-truth data. Results: The target registration errors (TREs) between the FM positions estimated by optical flow algorithms and the ground-truth ranged from 1.82 ± 1.05 to 1.98 ± 1.17 mm, which is within the uncertainty of the ground-truth position. Two algorithm groups, namely, optical flow and demons, were used to estimate tumor positions with TREs ranging from 1.29 ± 1.21 to 1.78 ± 1.75 mm. With respect to the deformed position for tumors, for the 2 DIR algorithm groups, the maximum differences of the deformed positions for gross tumor volume tracking were approximately 4.55 to 7.55 times higher than the mean differences. Errors caused by the aforementioned difference in the Hounsfield unit values were also observed. Conclusions: We quantitatively evaluated 10 DIR algorithms using 4D CT images of 5 patients and compared the results with ground-truth data. The optical flow algorithms showed reasonable FM-tracking results in patient 4D CT images. The iterative optical flow method delivered the best performance in this study. With respect to the tumor volume, the optical flow and demons algorithms delivered the best performance.

An Optical Flow SBAS Technique for Glacier Surface Velocity Extraction Using SAR Images

The pixel offset-tracking (PO) technique developed from correlation matching for use on synthetic aperture radar (SAR) images has been widely employed to monitor glacier dynamics. However, the decorrelation caused by rapid changes in a glacier surface reduces the integrity of flow velocity extraction. In this letter, we propose a novel method, termed the optical flow (OF) small baseline subset(SBAS), developed from the optical flow algorithm, which is defined as the apparent motion of individual pixels on the image plane. The OF algorithm can compute dense flows at the individual pixel level with a low computational cost and may serve as an alternative to PO for estimating the glacier velocity field. During processing, We selected the image pairs having short spatiotemporal baselines and calculated their offset series according to the OF algorithm. We then used an interval estimation strategy to eliminate outliers to refine stacked offsets. Finally, the least-squares method was used to calculate the displacement series for each pixel. We tested the proposed method utilizing eight ALOS-2/PALSAR-2 images on a temperate debris-covered glacier, the Hailuogou Glacier on the southeastern Tibetan Plateau. Compared with PO-SBAS, our method effectively improves the coverage of glacier flow velocity monitoring from 73.5% to 99.6%.

Method for Forecasting Ionospheric Electron Content Fluctuations Based on the Optical Flow Algorithm

We present the optical flow algorithm for forecasting the rate of total electron content index (OFROTI). It consists of a method for predicting maps of rapid fluctuations of ionospheric electron content in terms of global navigation satellite system (GNSS) dual-frequency phase measurements of the rate of change of total electron content index (ROTI). The forecast is made in space and time, at horizons up to more than 6 h. These forecast maps will consist of the ROTI spatial distribution in the northern hemisphere above 45° latitude. The prediction method models the ROTI spatial distribution as a pseudoconservative flux, i.e., exploiting the inertia of the flux of ROTI to determine the future position. This idea is implemented as a modification of the optical flow image processing technique. The algorithm has been modified to deal with the nonconservation of the ROTI quantity in time. We show that the method can predict both, the local value of ROTI and also the regions with ROTI above a given level, better than the prediction using the current map as forecast, i.e., predicting by a current map from horizons of 15 min up to 6 h. The method was tested on 11 representative active and calm days during 2015 and 2018 from the multi-GNSS (GPS, GLONASS, Galileo, and Beidou) multifrequency measurements of more than 250 multi-GNSS receivers above 45°N latitude, including the high rate (1 Hz) measurements of Greenland geodetic network (GNET) network among the International GNSS Service network.

Preliminary study to investigate the applicability of optical strain measurement technique for the detection of hot cracks in laser metal deposited layers

Laser metal deposition (LMD) as an additive manufacturing technique became increasingly important in recent years and thus the demand for component safety. This is the reason, for the need for reliable in-situ defect detection techniques. For laser beam weld seams an optical measurement technique based on an optical flow algorithm was successfully used to define the critical straining conditions that lead to hot cracking. This algorithm was adapted for bead-on-plate weld seams on LMD deposited layers of IN718 alloy while performing external strain on the specimen in an externally loaded hot cacking test facility. The resulting transversal hot cracks along the weld seam were localized via X-Ray inspection and the type of cracking confirmed by Scanning Electron Microscopy (SEM). The strain distribution was measured in the vicinity of the solidification front and correlated to the detected hot cracks. Based on the results this technique could be adopted for LMD experiments.

Synthetic velocity measurement algorithm of monocular vision based on square-root cubature Kalman filter.

The velocity measurement algorithm based on vision is widely used in unmanned aerial vehicle navigation. Under uneven illumination intensity distribution, the traditional Lucas-Kanade (LK) optical flow (OF) algorithm has problems arising from low computational accuracy and poor adaptability. To solve these problems, we propose a monocular vision integrated velocity measurement system based on the square-root cubature Kalman filter (SRCKF). The LK OF and the optimized oriented FAST and rotated BRIEF (ORB) algorithms are used to process the visual information obtained using a camera. The SRCKF algorithm is tasked with fusing the LK OF and optimized ORB information, thereby improving the accuracy of velocity and alleviating the sensitivity of the LK OF to variations in illumination conditions. Finally, an outdoor unmanned aerial vehicle flight test was undertaken. The experimental results show that the proposed method provides an accurate measurement of the velocity in variable illumination environments.

Fast Event-Based Optical Flow Estimation by Triplet Matching

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\gt$</tex-math></inline-formula> 10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.

Anchor Points Based Accurate Fault Locating in Large-Scale Photovoltaic Plants via Aerial Infrared Videos

For large-scale photovoltaic plants, accurate fault locating provides strong support for efficient maintenance. Based on aerial infrared videos, a progressive locating method is proposed to locate faults in the photovoltaic plants. The photovoltaic string is first located, and the faults are located on the modules of the corresponding photovoltaic string. For the former, the image extracted from the infrared video is binarized in HSV color space, and then the target pixels are horizontally and vertically projected to realize photovoltaic string segmentation and locating. For the latter, the faults are first located in the local coordinate system of each frames and then located in the overall coordinate system of the video by conversion. The conversion is realized by the matching of the anchor points in adjacent frames. The anchor points are detected by AKAZE algorithm and the matching is achieved by Lucas–Kanade sparse optical flow algorithm. The overall coordinate expression of the fault is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(R, C, (x, y))$</tex-math></inline-formula> that consists of the row index number <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</i> of the photovoltaic string in the photovoltaic plant, the column index number <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$C$</tex-math></inline-formula> of the photovoltaic string in the current row <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> , and the faulty module location <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(x, y)$</tex-math></inline-formula> on the photovoltaic string. Infrared videos of a 30-MW photovoltaic plant are tested, and the average accuracy of locating was 97.5%. Compared with nonprogressive locating method, the average accuracy of locating is higher by 17.5%. It can be seen that the proposed progressive locating method can accurately locate faults in large-scale photovoltaic plants.

Unsupervised Learning of Auroral Optical Flow for Recognition of Poleward Moving Auroral Forms

Poleward moving auroral forms (PMAFs) are one of the most common dayside auroral phenomena, which are important to study the dynamics of the polar cusp. In this article, we propose an unsupervised auroral optical flow (UnAurFlow) model for recognizing the PMAFs from auroral image sequences captured by all-sky imagers. Since auroral shape and illumination change continuously and randomly during the evolution, and auroral motion violates the brightness constancy assumption, UnAurFlow designs an auroral deformation detection module based on bidirectional optical flow and exploits the census transform to compensate for the auroral luminosity changes. UnAurFlow is trained in an unsupervised manner, which circumvents the difficulty of manually labeling auroral optical flow. The validity of UnAurFlow is qualitatively and quantitatively evaluated using auroral observations in 2003–2005 at the Arctic Yellow River Station. Based on the auroral optical flow computed by UnAurFlow, and combining with ResNet and the attention mechanism, PMAFs are automatically recognized with a precision of 84.77% from continuous auroral observations, which outperform the state-of-the-art unsupervised optical flow algorithms. The temporal occurrence distribution of PMAFs is prenoon–postnoon asymmetric under negative interplanetary magnetic field (IMF) By conditions, which is in good agreement with the previous statistical study.

Gray consistency optical flow algorithm based on mask-R-CNN and a spatial filter for velocity calculation.

The optical flow method has been widely used to measure the vehicle velocity by observing the stationary ground with a camera looking-down. However, when there are moving objects on the stationary ground, the interfering optical flow field will be generated, which decreases the velocity measurement accuracy of a vehicle relative to the ground. In order to reduce the effects caused by moving objects, this paper integrates pyramid Lucas-Kanade (LK) algorithm with the gray consistency method to use the information of color images thoroughly. First, a mask region with convolutional neural network (Mask-R-CNN) is used to recognize the objects that have motions relative to the ground, and it covers them with masks to enhance the similarity between pixels and to reduce the impacts of the noisy moving pixels. Then images are decomposed into three channels, red, green, and blue (i.e., R, G, and B), and processed by median filter. Based on the gray consistency method, the optical flow can be obtained by the pyramid LK algorithm. Finally, the velocity is calculated by the optical flow value. The prominent advantages of the proposed algorithm are: (i) increase the velocity measurement accuracy of a vehicle relative to the ground; (ii) use the information of color images acquired with cameras thoroughly and obtain velocity calculation outputs with less fluctuation; (iii) reduce wrong values caused by noises that are from the origin image and introduced by similar color masks. Four experiments are conducted to test the proposed algorithm and results with superior precision and reliability show the feasibility and effectiveness of the proposed method for the velocity measurement accuracy of a vehicle relative to the ground.

Uncertainty Estimation of Dense Optical Flow for Robust Visual Navigation.

This paper presents a novel dense optical-flow algorithm to solve the monocular simultaneous localisation and mapping (SLAM) problem for ground or aerial robots. Dense optical flow can effectively provide the ego-motion of the vehicle while enabling collision avoidance with the potential obstacles. Existing research has not fully utilised the uncertainty of the optical flow—at most, an isotropic Gaussian density model has been used. We estimate the full uncertainty of the optical flow and propose a new eight-point algorithm based on the statistical Mahalanobis distance. Combined with the pose-graph optimisation, the proposed method demonstrates enhanced robustness and accuracy for the public autonomous car dataset (KITTI) and aerial monocular dataset.

Out-of-plane displacement measurement for an optical fiber image transmission element using optical flow.

To realize the industrial in-situ deformation detection of an optical fiber image transmission element, a method for out-of-plane displacement detection based on the combination of optical flow and an image spherizing algorithm is presented. After capturing the 2D deformation images, the out-of-plane displacement can be extracted from the in-plane displacement estimated by the optical flow method, which means that the out-of-plane displacement information contained in a 2D image is fully used. The principle of the Brox optical flow method is introduced and the accuracy of the method is proven by comparing test results with the digital image correlation method. Experimental results show that the relative errors and the absolute errors of the measurement are less than 2% and 0.1 µm. Because the optical flow algorithm introduces a time parameter and the smoothness assumption, the new method is more appropriate for dynamic and microscale out-of-plane displacement measurement. Unlike conventional methods, it does not need phase unwrapping or frequency domain conversion during detection and has many attractive characteristics such as nondestructive testing, a simple arrangement, and low errors. This method has been applied to the industrial in-situ monitoring of an optical fiber image transmission element.

Application of Optical Flow Technique and Photogrammetry for Rockfall Dynamics: A Case Study on a Field Test

Optical flow is a vision-based approach that is used for tracking the movement of objects. This robust technique can be an effective tool for determining the source of failures on slope surfaces, including the dynamic behavior of rockfall. However, optical flow-based measurement still remains an issue as the data from optical flow algorithms can be affected by the varied photographing environment, such as weather and illuminations. To address such problems, this paper presents an optical flow-based tracking algorithm that can be employed to extract motion data from video records for slope monitoring. Additionally, a workflow combined with photogrammetry and the optical flow technique has been proposed for producing highly accurate estimations of rockfall motion. The effectiveness of the proposed approach has been evaluated with the dataset obtained from a photogrammetry survey of field rockfall tests performed by the authors in 2015. The results show that the workflow adopted in this study can be suitable to identify rockfall events overtime in a slope monitoring system. The limitations of the current approach are also discussed.

Wind Flow Characterisation over a PV Module through URANS Simulations and Wind Tunnel Optical Flow Methods

Despite their simplicity, photovoltaic (PV) modules are often arranged in structures that can be affected by severe and complex wind loads: in this context, the wind flow and the dynamic excitation induced by vortex shedding can introduce unexpected aeroelastic responses. This work introduces a novel wind tunnel application of experimental techniques to address this issue by the use of flow visualisation and video postprocessing, through the optical flow algorithm. Numerical simulations based on unsteady Reynolds-averaged Navier–Stokes (RANS) models are performed and compared against the experimental wind tunnel tests on a PV panel that was also instrumented with pressure taps. A setup with a 65∘ tilt angle was examined because, based on preliminary analyses, it was considered interesting for the free flow–wake transition associated with the dynamic response of the PV panel. The comparison of the experimental and numerical average wind fields supported that the proposed optical flow method was appropriate for characterising the wake of the panel, because there was enough seeding to perform the video postprocessing. Experiments and numerical predictions were compared as regards the average pressure distribution on the panel surfaces, and the average percentage was in the error of 7%; this supports that the URANS method was capable of reproducing the average behaviour of the panel, as well as for the selected configuration, which is particularly challenging. Furthermore, the simulated and measured power spectral densities of the wind speed were compared, and this resulted in the numerical model quite faithfully reproducing the frequency of the peak at 5 m/s, while the error was in the order of 20% for the 10 m/s case; this supports that, despite the URANS approach being affected by well-known critical points regarding the simulation of instantaneous quantities, it can be employed to elaborate information that can be particularly useful for the structural design of the panel. This kind of result can be considered as a first step, obtained with simplified and affordable methods, towards a characterisation of the dynamic behaviour of a PV panel in a real-world setup.