Optical Flow Algorithm Research Articles

3D wind field profiles from hyperspectral sounders: revisiting optic-flow from a meteorological perspective

In this work, we present an efficient optic flow algorithm for the extraction of vertically resolved 3D atmospheric motion vector (AMV) fields from incomplete hyperspectral image data measures by infrared sounders. The model at the heart of the energy to be minimized is consistent with atmospheric dynamics, incorporating ingredients of thermodynamics, hydrostatic equilibrium and statistical turbulence. Modern optimization techniques are deployed to design a low-complexity solver for the energy minimization problem, which is non-convex, non-differentiable, high-dimensional and subject to physical constraints. In particular, taking advantage of the alternate direction of multipliers methods (ADMM), we show how to split the original high-dimensional problem into a recursion involving a set of standard and tractable optic-flow sub-problems. By comparing with the ground truth provided by the operational numerical simulation of the European Centre for Medium-Range Weather Forecasts (ECMWF), we show that the performance of the proposed method is superior to state-of-the-art optical flow algorithms in the context of real infrared atmospheric sounding interferometer (IASI) observations.

Multi-Target Rumination Behavior Analysis Method of Cows Based on Target Detection and Optical Flow Algorithm

Rumination behavior is closely associated with factors such as cow productivity, reproductive performance, and disease incidence. For multi-object scenarios of dairy cattle, ruminant mouth area images accounted for little characteristic information, which was first put forward using an improved Faster R-CNN target detection algorithm to improve the detection performance model for the ruminant area of dairy cattle. The primary objective is to enhance the model’s performance in accurately detecting cow rumination regions. To achieve this, the dataset used in this study is annotated with both the cow head region and the mouth region. The ResNet-50-FPN network is employed to extract the cow mouth features, and the CBAM attention mechanism is incorporated to further improve the algorithm’s detection accuracy. Subsequently, the object detection results are combined with optical flow information to eliminate false detections. Finally, an interpolation approach is adopted to design a frame complementary algorithm that corrects the detection frame of the cow mouth region. This interpolation algorithm is employed to rectify the detection frame of the cow’s mouth region, addressing the issue of missed detections and enhancing the accuracy of ruminant mouth region detection. To overcome the challenges associated with the inaccurate extraction of small-scale optical flow information and interference between different optical flow information in multi-objective scenes, an enhanced GMFlowNet-based method for multi-objective cow ruminant optical flow analysis is proposed. To mitigate interference from other head movements, the MeanShift clustering method is utilized to compute the velocity magnitude values of each pixel in the vertical direction within the intercepted ruminant mouth region. Furthermore, the mean square difference is calculated, incorporating the concept of range interquartile, to eliminate outliers in the optical flow curve. Finally, a final filter is applied to fit the optical flow curve of the multi-object cow mouth movement, and it is able to identify rumination behavior and calculate chewing times. The efficacy, robustness, and accuracy of the proposed method are evaluated through experiments, with nine videos capturing multi-object cow chewing behavior in different settings. The experimental findings demonstrate that the enhanced Faster R-CNN algorithm achieved an 84.70% accuracy in detecting the ruminant mouth region, representing an improvement of 11.80 percentage points over the results obtained using the Faster R-CNN detection approach. Additionally, the enhanced GMFlowNet algorithm accurately identifies the ruminant behavior of all multi-objective cows, with a 97.30% accuracy in calculating the number of ruminant chewing instances, surpassing the accuracy of the FlowNet2.0 algorithm by 3.97 percentage points. This study provides technical support for intelligent monitoring and analysis of rumination behavior of dairy cows in group breeding.

Divisive normalization processors in the early visual system of the Drosophila brain

Divisive normalization is a model of canonical computation of brain circuits. We demonstrate that two cascaded divisive normalization processors (DNPs), carrying out intensity/contrast gain control and elementary motion detection, respectively, can model the robust motion detection realized by the early visual system of the fruit fly. We first introduce a model of elementary motion detection and rewrite its underlying phase-based motion detection algorithm as a feedforward divisive normalization processor. We then cascade the DNP modeling the photoreceptor/amacrine cell layer with the motion detection DNP. We extensively evaluate the DNP for motion detection in dynamic environments where light intensity varies by orders of magnitude. The results are compared to other bio-inspired motion detectors as well as state-of-the-art optic flow algorithms under natural conditions. Our results demonstrate the potential of DNPs as canonical building blocks modeling the analog processing of early visual systems. The model highlights analog processing for accurately detecting visual motion, in both vertebrates and invertebrates. The results presented here shed new light on employing DNP-based algorithms in computer vision.

Brain-like position measurement method based on improved optical flow algorithm

In this paper, a brain-like navigation scheme based on fuzzy kernel C-means (FKCM) clustering assisted pyramid Lucas Kanade (LK) optical flow algorithm is developed to measure the position of vehicle. The Speed Cell and Place Cell in animals’ brain are introduced to construct the brain-like navigation mechanism which involves the optical flow method and image template matching to imitate the cells above-mentioned separately. To eliminate the singular values during optical flow calculation, the output of pyramid LK algorithm is clustered by FKCM algorithm firstly. Then, the velocity is calculated and integrated to get the position of the vehicle, and the brain-like navigation scheme is introduced to correct the position measurement errors by eliminating the accumulated errors resulting from velocity integration. The prominent advantages of the presented method are: (i) a pure visual brain-like position measurement method based on the concept of speed cells and place cells is proposed, making visual navigation more accurate and intelligent; (ii) the FKCM algorithm is used to eliminate the singular value of the pyramid LK algorithm, which improves the calculated velocity accuracy. Also, experimental comparison with classical pyramid LK algorithm is given to illustrate the superiority of the proposed method in position measurement.

A Computer Vision-Based Respiratory Rate Monitoring and Alarm System

Breathing rate is one of the most important vital signals for monitoring health status and reflecting conditions of dangerous diseases. Previous contactless breath monitoring methods were more convenient than contact methods, but they were not suitable for the actual sleeping environment because of the narrow field of vision (FoV). This study proposed a breathing rate monitoring strategy using a mono camera to track and detect sleep apnea phenomena. Breathing rates were first tracked among consecutive image frames. The human body area was then isolated and magnified using a deep neural network (DNN) model before applying the optical flow algorithm to extract and monitor the up and down changes caused by respiration. The most variated directions of the body feature’s motions were detected based on the Principal Component Analysis (PCA) method. Breathing rate was the number of times the signal amplitude peaks per minute. The comparison between predicted values and manually estimated was used for evaluating the accuracy of the method. The accuracy of our method in various light, position, and distance conditions is 2 breaths/minute (<10%) for children and less than 1 breath/minute (<5%) for adults. The study has two main contributions: (1) monitoring breathing rate at home gives comfortable feelings to patients and caregivers, expanding the potential of applying modern technology to clinics, (2) the study has solved the problem of tracking small movements in videos with relatively large FoV in real-time. Perspectively, we will be employed the method in a home-based respiratory rate monitoring system.

Dynamic 3D phase-shifting profilometry based ona corner optical flow algorithm.

Real-time 3D reconstruction has been applied in many fields, calling for many ongoing efforts to improve the speed and accuracy of the used algorithms. Phase shifting profilometry based on the Lucas-Kanade optical flow method is a fast and highly precise method to construct and display the three-dimensional shape of objects. However, in this method, a dense optical flow calculation is required for the modulation image corresponding to the acquired deformed fringe pattern, which consumes a lot of time and affects the real-time performance of 3D reconstruction and display. Therefore, this paper proposes a dynamic 3D phase shifting profilometry based on a corner optical flow algorithm to mitigate this issue. Therein, the Harris corner algorithm is utilized to locate the feature points of the measured object, so that the optical flow needs to calculate for only the feature points which, greatly reduces the amount of calculation time. Both our experiments and simulations show that our method improves the efficiency of pixel matching by four times and 3D reconstruction by two times.

Individuality of a group: detailed walking ability analysis of broiler flocks using optical flow approach

Impaired walking ability is one of the most important factors affecting broiler welfare. Routine monitoring of walking ability provides insights in the welfare status of a flock and assists farmers in taking remedial measures at an early stage. Several computer vision techniques have been developed for automated assessment of walking ability, providing an objective and biosecure alternative to the currently more subjective and time-consuming manual assessment of walking ability. However, these techniques mainly focus on assessment of averages at flock level using pixel movement. Therefore, the aim of this study was to investigate the potential of optical flow algorithms to identify flock activity, distribution and walking ability in a commercial setting on levels close to individual monitoring. We used a combination of chicken segmentation and optical flow methods, where chicken contours were first detected and were then used to identify activity, spatial distribution, and gait score distribution (i.e. walking ability) of the flock via optical flow. This is a step towards focusing more on individual chickens in an image and its pixel representation. In addition, we predicted the gait score distribution of the flock, which is a more detailed assessment of broiler walking ability compared to average gait score of the flock, as slight changes in walking ability are more likely to be detected when using the distribution compared to the average score. We found a strong correlation between predicted and observed gait scores (R2 = 0.97), with separate gait scores all having R2 > 0.85. Thus, the algorithm used in this study is a first step to measure broiler walking ability automatically in a commercial setting on a levels close to individual monitoring. These validation results of the developed automatic monitoring of flock activity, distribution and gait score are promising, but further validation is required (e.g. for chickens at a younger age, with very low and very high gait scores).

Flight state recognition for UAV optical flow velocity measurement

For the visual navigation and positioning of drones, the optical flow algorithm is used to calculate the flight speed of the drone, the calculation errors of its rotation state, and the vertical take-off and landing state. Based on the characteristics of its optical flow data, this paper compares various classification calculation methods. It uses the median KNN curve of the optical flow vector to simplify its feature quantity. Finally, it realizes the motion of the UAV with the optical flow characteristics based on the SVM algorithm. State recognition experiments have proved that this method can effectively identify the motion state of the UAV, and the calculation efficiency is high.

Optical Flow-Based Full-Field Quantitative Blood-Flow Velocimetry Using Temporal Direction Filtering and Peak Interpolation.

The quantitative measurement of the microvascular blood-flow velocity is critical to the early diagnosis of microvascular dysfunction, yet there are several challenges with the current quantitative flow velocity imaging techniques for the microvasculature. Optical flow analysis allows for the quantitative imaging of the blood-flow velocity with a high spatial resolution, using the variation in pixel brightness between consecutive frames to trace the motion of red blood cells. However, the traditional optical flow algorithm usually suffers from strong noise from the background tissue, and a significant underestimation of the blood-flow speed in blood vessels, due to the errors in detecting the feature points in optical images. Here, we propose a temporal direction filtering and peak interpolation optical flow method (TPIOF) to suppress the background noise, and improve the accuracy of the blood-flow velocity estimation. In vitro phantom experiments and in vivo animal experiments were performed to validate the improvements in our new method.

Optical Flow Glottovibrogram for the examination of vocal fold pathology.

Laryngeal high-speed video endoscopy is performed to examine the cycles of vocal fold vibrations in detail and to diagnose voice abnormalities. One of the recent image processing techniques for visualizing vocal fold vibration is optical flow-based playbacks, which include optical flow kymograms (OFKG) for local dynamics, optical flow glottovibrogram (OFGVG) and glottal optical flow waveforms (GOFW) for global dynamics. In recent times, various optical flow computing algorithms have been developed. In this paper, we used four well-known optical flow algorithms Horn Schunk, Lucas Kanade, Gunnar Farneback, and TVL1 to construct the optical flow playbacks. The proposed playback reliability is examined by comparing them to traditional representations such as Phonovibrogram (PVG). Since PVG and OFGVG are interconnected, a comparison study was carried out to better comprehend their interaction.Clinical Relevance- Both OFGVG and PVG add to the precision of interpreting pathological conditions by offering complementary information to the conventional spatiotemporal representations.

Background oriented schlieren image displacement estimation method based on global optical flow

Background oriented schlieren (BOS) technology is a recent non-contact optical diagnostic technology, which can be used for the visualization of variable density flow and the quantitative measurement of the refractive index field. The accuracy of BOS image displacement estimation directly determines the accuracy of refractive index field reconstruction. For BOS images, the optical path construction method, vibration and uneven illumination, blurring and defocusing effects, and the refocusing effect of volume refraction make it difficult for the BOS image sequence to satisfy the constant brightness assumption. In this study, the optical flow algorithm based on the assumption of a constant gradient in PIV is improved and applied to BOS image displacement estimation. A data item based on the assumption of constant brightness combined with the assumption of constant gradient is proposed. The first-order div-curl regularization constraint is used as the spatial smoothing condition, and the non-convex Charbonnier penalty function is used to determine the displacement field by the variational method to minimize the energy function. The proposed method is tested on the synthetic images obtained by two-dimensional turbulent direct numerical simulation (DNS), the BOS synthetic images with brightness changes, and the actual experimental BOS images of the flow field above a premixed butane/air combustion flame with different equivalence ratios. Compared with the current advanced optical flow algorithm, this method has higher accuracy and robustness. Overall, the proposed method can not only obtain more accurate results when the brightness of the BOS synthetic image and experimental image changes, but also helps in retaining the flow details such as small divergence and vorticity structure of the heat flow field while reducing the outliers.

Long-term liver lesion tracking in contrast-enhanced ultrasound videos via a siamese network with temporal motion attention.

Propose: Contrast-enhanced ultrasound has shown great promises for diagnosis and monitoring in a wide range of clinical conditions. Meanwhile, to obtain accurate and effective location of lesion in contrast-enhanced ultrasound videos is the basis for subsequent diagnosis and qualitative treatment, which is a challenging task nowadays. Methods: We propose to upgrade a siamese architecture-based neural network for robust and accurate landmark tracking in contrast-enhanced ultrasound videos. Due to few researches on it, the general inherent assumptions of the constant position model and the missing motion model remain unaddressed limitations. In our proposed model, we overcome these limitations by introducing two modules into the original architecture. We use a temporal motion attention based on Lucas Kanade optic flow and Karman filter to model the regular movement and better instruct location prediction. Moreover, we design a pipeline of template update to ensure timely adaptation to feature changes. Results: Eventually, the whole framework was performed on our collected datasets. It has achieved the average mean IoU values of 86.43% on 33 labeled videos with a total of 37,549 frames. In terms of tracking stability, our model has smaller TE of 19.2 pixels and RMSE of 27.6 with the FPS of 8.36 ± 3.23 compared to other classical tracking models. Conclusion: We designed and implemented a pipeline for tracking focal areas in contrast-enhanced ultrasound videos, which takes the siamese network as the backbone and uses optical flow and Kalman filter algorithm to provide position prior information. It turns out that these two additional modules are helpful for the analysis of CEUS videos. We hope that our work can provide an idea for the analysis of CEUS videos.

Comparison of schlieren-based techniques for measurements of a turbulent and wavy free surface

The free surface at an air–water interface can provide information regarding bathymetric complexities, as well as the subsurface flow. We present a comparison of the performance of two recent advances in light-based optical techniques for free surface measurements, total internal reflection-deflectometry and moon-glade background-oriented schlieren, with the more established method of free surface synthetic schlieren. We make use of an optical flow algorithm over the more traditional digital image correlation, in order to obtain higher spatial resolution data across the imaged free surface domain. The optical flow algorithm presents additional benefits, such as computational efficiency and robustness in capturing large displacements and straining of tracked features. The three optical techniques are assembled in synchronization to image two free surface conditions: (1) a free surface being impinged upon by an underlying turbulent, free-shear flow and (2) a random and irregular wave field induced by a free jet. Using the high-resolution measurements, we provide insight on the emergence of multiple free surface dynamics for a turbulent free surface. We present a comprehensive discussion on the benefits and drawbacks of each technique, including suggestions on the suitability of each technique for several experimental constraints.Graphical abstract

Electrographic flow mapping guided catheter ablation offers advantages for patients with persistent atrial fibrillation.

Catheter ablation (CA) remains challenging due to suboptimal success rates in persistent atrial fibrillation (AF). Existing mapping technologies cannot reliably distinguish sources in this patient population. Recently, the novel electrographic flow (EGF) mapping system was developed using a modified Horn-Schunk optical flow algorithm to detect and quantify patterns of electrical wavefront propagation in the atria. To test the hypothesis that targeted source ablation based on EGF mapping is superior to empiric AF ablation. We included all consecutive patients undergoing EGF guided ablation for persistent AF. All patients underwent pulmonary vein isolation (PVI) and were treated with the same EAM system (CARTO). The outcome of PVI+EGF guided CA was compared with data of PVI-only procedures (PVI-only group) and PVI plus additional empiric adjunctive linear and substrate ablations (PVI+LINES group). 12-months outcome as freedom from AF and atrial tachycardia/flutter (AT/AFL), procedural safety and efficiency characterized by procedure duration, fluoroscopy use, radiofrequency applications and duration, were analyzed. Both intention-to-treat and per protocol analysis were conducted. A total number of 70 patients (39 in PVI+EGF, 16 in PVI-only and 15 patients in PVI+LINES group) were enrolled. Intention-to-treat analysis showed fewer AF recurrences in PVI+EGF as compared with the PVI-only or PVI+LINES groups at 12 months (25.6%vs. 62.5%vs. 53.3%, p=.02). There were no differences in AT/AFL recurrence (17.9%vs. 37.5%vs. 20.0%, p=.37). Procedure times were longer in PVI+EGF group (p<.01), and there were no differences in fluoroscopy use (p=.67). Our data suggest that patients treated with EGF-guided CA developed fewer AF recurrences. Although the procedure times are longer, it seems to be safe and offers a more targeted, patient-specific ablation strategy beyond PVI than adjunctive empiric lines and substrate ablation in this complex group of patients.

An investigation of camera movements and capture techniques on optical flow for real-time rendering and presentation

New and interesting uses for portable devices include the creation and viewing of 3D models and 360-degree photos of real landscapes. To provide a 3D model and a 360-degree view of a scenario, these apps search for real-time rendering and presentation. This study examines the impact of real-time image processing on movements in camera view and the application of optical flow algorithms. The optical flows are affected by how the objects involved motion. The way the image is recorded and the camera movements affect the relative motion. As a consequence, camera motions are responsive to optical flow algorithms. To record an image, the camera may pan around, move in a straight path, or move randomly. We have captured datasets of videos produced in different contexts to better replicate real-world scenarios. Each dataset was captured in a variety of illumination situations, camera movements, and indoor and outdoor recording sites. Here, to determine the most effective optical flow algorithms for use in near real-time applications, such as augmented reality and virtual reality, we compare results based on quality and processing delay for each video frame. We conducted a comparison study to better understand how random camera motion affects real-time video processing. These methods can be used to handle a variety of real-world issues, such as object tracking, video segmentation, structure from motion, gesture tracking, and so on.

Measuring landslide-driven ground displacements with high-resolution surface models and optical flow

This study proposes and evaluates the performance of two image matching algorithms applied to hillshades derived from consecutive high-resolution digital surface models (DSMs) to measure surface displacements on landslides. The method is applied on Te Horo, a slow-moving landslide in the Dart Valley of New Zealand’s Southern Alps/Kā Tiritiri o te Moana using pairs of hillshades derived from Airborne and Satellite Photogrammetric Mapping (APM/SPM) of imagery captured in 2018 and 2020. This novel approach uses the consistency of displacement predictions generated from multiple hillshade pairs to gauge match quality and mask unreliable displacement predictions. The study compares a widely used normalised cross-correlation algorithm (NCC) alongside an optical flow approach for image matching. The performances of both algorithms are assessed against manually derived displacements of prominent surface features. We demonstrate the effectiveness of the masking approach as well as the good performance of the optical flow algorithm in delivering dense, accurate displacement measurements efficiently, particularly when high resolution DSMs are available. The results show that the main translational body of the landslide was displaced at rates averaging 25–55 mm day−1 over the 2018–2020 period.

Application of 1,000 fps High-Speed Angiography to In-Vitro Hemodynamic Evaluation of Left Ventricular Assist Device Outflow Graft Configurations.

Left ventricular assist device (LVAD)-induced hemodynamics are characterized by fast-moving flow with large variations in velocity, making quantitative assessments difficult with existing imaging methods. This study demonstrates the ability of 1,000 fps high-speed angiography (HSA) to quantify the effect of the surgical implantation angle of a LVAD outflow graft on the hemodynamics within the ascending aorta in vitro . High-speed angiography was performed on patient-derived, three-dimensional-printed optically opaque aortic models using a nonsoluble contrast media, ethiodol, as a flow tracer. Outflow graft configuration angles of 45° and 90° with respect to the central aortic axis were considered. Projected velocity distributions were calculated from the high-speed experimental sequences using two methods: a physics-based optical flow algorithm and tracking of radio-opaque particles. Particle trajectories were also used to evaluate accumulated shear stress. Results were then compared with computational fluid dynamics (CFD) simulations to confirm the results of the high-speed imaging method. Flow patterns derived from HSA coincided with the impingement regions and recirculation zones formed in the aortic root as seen in the CFD for both graft configurations. Compared with the 45° graft, the 90° configuration resulted in 81% higher two-dimensional-projected velocities (over 100 cm/s) along the contralateral wall of the aorta. Both graft configurations suggest elevated accumulated shear stresses along individual trajectories. Compared with CFD simulations, HSA successfully characterized the fast-moving flow and hemodynamics in each LVAD graft configuration in vitro , demonstrating the potential utility of this technology as a quantitative imaging modality.

Application of optical flow algorithm for drift correction in electron microscopy images.

Transmission electron microscopy (TEM) image drift correction has been effectively addressed using diverse approaches, including the cross correlation algorithm (CC) and other strategies. However, most of the strategies fall short of achieving sufficient accuracy or cannot strike a balance between time consumption and accuracy. The present study proposes a TEM image drift correction strategy that enhances accuracy without any additional time consumption. Unlike the CC algorithm that matches pixels one by one, our approach involves the extraction of multiple feature points from the first TEM image and then uses the Lucas-Kanade (LK) optical flow algorithm to calculate the optical field of these feature points in the subsequent TEM images. The LK algorithm is used to calculate the instantaneous velocity of these feature points, which can help track the movement of the TEM image series. In addition, a high-precision sub-pixel level correction strategy by the utilization of linear interpolation during the correction process is developed in this work. Experimental results confirm that this strategy offers superior accuracy in comparison with the CC algorithm and also is insensitive to the size of the image. Furthermore, we offer a semantic segmentation neural network for electron microscope image pre-processing, thereby expanding the applicability of our methodology.

SecureCam: Selective Detection and Encryption Enabled Application for Dynamic Camera Surveillance Videos

Using dynamic surveillance cameras for security has significantly increased the privacy concerns for captured individuals. Malicious users may misuse these videos by performing Replay and/or Man-in-the-Middle attacks during storage or recording over the network. Considering these risks, this paper proposes an effective security application SecureCam based on selective detection (focused moving objects) and protection using encryption. For object detection, this paper implements a novel low computational unsupervised learning algorithm i.e. Motion-Fusion (MF) for more precise motion detection in the mobile camera videos. After that, selective encryption (SE) is applied by the lightweight Chacha20 cipher to the detected video parts. Proposed SecureCam is extensively evaluated based on performance analysis, security analysis and computational complexity. For object detection, the comparative evaluation shows that the MF algorithm outperforms traditional state-of-the-art dense optical flow (DOF) algorithm with an average (mean) difference increase: in the accuracy of 54%; and in the precision of 42% making it computationally effective for such videos. The visual results with 21% encryption space ratio (ESR) indicate that the videos are sufficiently protected against identification. Overall comparative evaluation with existing approaches also affirm the significance and utility of proposed SecureCam for internet of multimedia things (IoMT) environment.

Performance of Background Oriented Schlieren with different background patterns and image processing techniques

Synthetic tests of the accuracy, spatial resolution and robustness with respect to large gradient of the apparent displacement are performed for Background Oriented Schlieren (BOS) measurements using various background patterns and image processing techniques, including multi-pass cross-correlation, iterative Horn–Schunck and Lucas–Kanade optical flow algorithms and Fourier Transform Profilometry (FTP). It is shown that FTP with periodic sinusoidal or binary fringe pattern outperforms all the other methods. With optimal pattern period λpattern=6−15pix it is able to resolve perturbations with wavelength down to λpattern2 and to obtain reliable results for images with displacement gradient up to 0.9 pix/pix. Better performance of FTP, compared to multi-pass cross-correlation, is confirmed for real experimental measurements of water temperature in natural convection near a heated vertical plate. FTP advantages and limitations, including possible nonlocal error, associated with phase unwrapping, are discussed.