Optical Flow Model Research Articles

Three-view cotton flower counting through multi-object tracking and RGB-D imagery

Monitoring the number of cotton flowers can provide important information for breeders to assess the flowering time and the productivity of genotypes because flowering marks the transition from vegetative growth to reproductive growth and impacts the final yield. Traditional manual counting methods are time-consuming and impractical for large-scale fields. To count cotton flowers efficiently and accurately, a multi-view multi-object tracking approach was proposed by using both RGB and depth images collected by three RGB-D cameras fixed on a ground robotic platform. The tracking-by-detection algorithm was employed to track flowers from three views simultaneously and remove duplicated counting from single views. Specifically, an object detection model (YOLOv8) was trained to detect flowers in RGB images and a deep learning-based optical flow model Recurrent All-pairs Field Transforms (RAFT) was used to estimate motion between two adjacent frames. The intersection over union and distance costs were employed to associate flowers in the tracking algorithm. Additionally, tracked flowers were segmented in RGB images and the depth of each flower was obtained from the corresponding depth image. Those flowers tracked with known depth from two side views were then projected onto the middle image coordinate using camera calibration parameters. Finally, a constrained hierarchy clustering algorithm clustered all flowers in the middle image coordinate to remove duplicated counting from three views. The results showed that the mean average precision of trained YOLOv8x was 96.4%. The counting results of the developed method were highly correlated with those counted manually with a coefficient of determination of 0.92. Besides, the mean absolute percentage error of all 25 testing videos was 6.22%. The predicted cumulative flower number of Pima cotton flowers is higher than that of Acala Maxxa, which is consistent with what breeders have observed. Furthermore, the developed method can also obtain the flower number distributions of different genotypes without laborious manual counting in the field. Overall, the three-view approach provides an efficient and effective approach to count cotton flowers from multiple views. By collecting the video data continuously, this method is beneficial for breeders to dissect genetic mechanisms of flowering time with unprecedented spatial and temporal resolution, also providing a means to discern genetic differences in fecundity, the number of flowers that result in harvestable bolls. The code and datasets used in this paper can be accessed on GitHub: https://github.com/UGA-BSAIL/Multi-view_flower_counting.

A simple optical flow model explains why certain object viewpoints are special.

A core challenge in perception is recognizing objects across the highly variable retinal input that occurs when objects are viewed from different directions (e.g. front versus side views). It has long been known that certain views are of particular importance, but it remains unclear why. We reasoned that characterizing the computations underlying visual comparisons between objects could explain the privileged status of certain qualitatively special views. We measured pose discrimination for a wide range of objects, finding large variations in performance depending on the object and the viewing angle, with front and back views yielding particularly good discrimination. Strikingly, a simple and biologically plausible computational model based on measuring the projected three-dimensional optical flow between views of objects accurately predicted both successes and failures of discrimination performance. This provides a computational account of why certain views have a privileged status.

A Variational Optical Flow Model for Accurate Motion Estimation from Rotational Image Sequences

A Variational Optical Flow Model for Accurate Motion Estimation from Rotational Image Sequences

Adaptive Fractional-Order Multi-Scale Optimization TV-L1 Optical Flow Algorithm

We propose an adaptive fractional multi-scale optimization optical flow algorithm, which for the first time improves the over-smoothing of optical flow estimation under the total variation model from the perspective of global feature and local texture balance, and solves the problem that the convergence of fractional optical flow algorithms depends on the order parameter. Specifically, a fractional-order discrete L1-regularization Total Variational Optical Flow model is constructed. On this basis, the Ant Lion algorithm is innovatively used to realize the iterative calculation of the optical flow equation, and the fractional order is dynamically adjusted to obtain an adaptive optimization algorithm with strong search accuracy and high efficiency. In this paper, the flexibility of optical flow estimation in weak gradient texture scenes is increased, and the optical flow extraction rate of target features at multiple scales is greatly improved. We show excellent recognition performance and stability under the MPI_Sintel and Middlebury benchmarks.

How to track and segment fish without human annotations: a self-supervised deep learning approach

Tracking fish movements and sizes of fish is crucial to understanding their ecology and behaviour. Knowing where fish migrate, how they interact with their environment, and how their size affects their behaviour can help ecologists develop more effective conservation and management strategies to protect fish populations and their habitats. Deep learning is a promising tool to analyse fish ecology from underwater videos. However, training deep neural networks (DNNs) for fish tracking and segmentation requires high-quality labels, which are expensive to obtain. We propose an alternative unsupervised approach that relies on spatial and temporal variations in video data to generate noisy pseudo-ground-truth labels. We train a multi-task DNN using these pseudo-labels. Our framework consists of three stages: (1) an optical flow model generates the pseudo-labels using spatial and temporal consistency between frames, (2) a self-supervised model refines the pseudo-labels incrementally, and (3) a segmentation network uses the refined labels for training. Consequently, we perform extensive experiments to validate our method on three public underwater video datasets and demonstrate its effectiveness for video annotation and segmentation. We also evaluate its robustness to different imaging conditions and discuss its limitations.

A Empirical Research on AI-Powered Athletic Posture Detection in Sports Training

Abstract: The current investigation delineates the efficacy of AI-facilitated detection of athletic postures within the realm of sports training. Employing a synthesis of literature review and empirical methodologies, data were amassed and scrutinized, affirming the study’s validity. The salient outcomes are manifold: (1) The frame difference algorithm efficaciously discerns inter-frame variances, evidencing pronounced adaptability and robustness, thereby enabling the recognition of weightlifting postures. (2) Confronting the challenge of negligible inter-frame disparities inherent in the frame difference algorithm, the research introduces a novel detection technique predicated on the cumulative inter-frame differences, which precisely pinpoints regions of posture alteration in weightlifting athletes. (3) Leveraging the dynamic space model of optical flow, the study ascertains the directional channel predicated on optical flow trajectory analyses, facilitating the identification of three distinct weightlifting postures: squatting, descending, and standing. (4) In alignment with the distinctive postural attributes of weightlifting athletes, a human posture paradigm was formulated, and a BP neural network classifier was deployed for both training and evaluative purposes, culminating in the successful differentiation of athlete from non-athlete entities within the training milieu. (5) The application of AI in posture recognition was extended to the scrutiny of pivotal postures and motions in weightlifting athletes, with experimental findings revealing a 98.21% accuracy rate in the recognition of force-exertion postures via the inter-frame difference method, and a flawless 100% accuracy in the identification of the apex and squatting postures. The enumeration of detected postures—encompassing knee extension, knee flexion, force application, squatting, and standing—through the poselet keyframe extraction approach, corresponded with the video count. Prospectively, AI’s role in athletic posture detection promises to augment coaches’ and athletes’ comprehension of their proficiencies and deficiencies, thereby steering training refinement and bolstering both the efficacy of training and the athletes’ caliber.

Freespace Optical Flow Modeling for Automated Driving

Freespace Optical Flow Modeling for Automated Driving

A framework for fluid motion estimation using a constraint-based refinement approach

Physics-based optical flow models have been successful in capturing the deformities in fluid motion arising from digital imagery. However, a common theoretical framework analyzing several physics-based models is missing. In this regard, we formulate a general framework for fluid motion estimation using a constraint-based refinement approach. We demonstrate that for a particular choice of constraint, our results closely approximate the classical continuity equation-based method for fluid flow. This closeness is theoretically justified by augmented Lagrangian method in a novel way. The convergence of Uzawa iterates is shown using a modified bounded constraint algorithm. The mathematical well-posedness is studied in a Hilbert space setting. Further, we observe a surprising connection to the Cauchy-Riemann operator that diagonalizes the system leading to a diffusive phenomenon involving the divergence and the curl of the flow. Several numerical experiments are performed and the results are shown on different datasets. Additionally, we demonstrate that a flow-driven refinement process involving the curl of the flow outperforms the classical physics-based optical flow method without any additional assumptions on the image data.

Particle image velocimetry combining unsupervised learning and optical flow model

Particle Image Velocimetry (PIV) is a multi-point, transient, contactless flow field measurement technique which can obtain accurate data information for the flow field. Particle image velocimetry technology based on deep learning has been widely used, with higher measurement accuracy and testing efficiency than traditional algorithms. Algorithms based on supervised learning need real data for network training. However, obtaining real flow field data is difficult, so unsupervised learning methods are more suitable for network training. The LiteFlowNet3 network has only four layers of flow inference, leading to limited reconstruction accuracy, especially for complex flow field. This paper adds one layer of flow inference based on the original network, while increasing the last layer of the loss. The network is trained using an unsupervised learning method, focusing on improving reconstruction accuracy for real-flow field training. The experimental results show that the improved model can reduce the average endpoint error by up to 30.89 % and 13.33 % compared to the original model and the unsupervised LiteFlowNet. In addition, by building a two-dimensional PIV system to simulate the advection and vortex field, real particle image pairs were obtained and put into the trained model for testing, proving the feasibility for obtaining velocity fields in real flow field.

Deep learning for particle image velocimetry with attentional transformer and cross-correlation embedded

Deep learning-powered methodologies have realized remarkable advancements in the fluid mechanics community, including applications in the particle image velocimetry (PIV) task. However, previous deep learning-based methods still lack robustness and generalization in the real flow scenarios. To solve this problem, we put forward a deep learning architecture called DeepST-CC for PIV estimation, which embeds an attentional transformer and cross-correlation strategy. Specifically, we introduce the Swin Transformer into the optical flow model Recurrent All-pairs Field Transforms (RAFT) to enhance the features of flow images. Then, the global matching between features is efficiently computed by applying a 4D correlation volume. Afterwards, the conventional cross-correlation method derives the initial velocity field from a coarse correlation, which is fed to the GRU-based flow update module. This approach enhances the robustness of the proposed model by incorporating coarse velocity information. Finally, a supervised learning strategy is performed to guide the model training on the synthetic dataset. Extensive experimental are conducted to demonstrate that our proposed approach delivers exceptional performance on the public dataset. In addition, DeepST-CC exhibits good generalization ability towards complex experimental PIV images.

Enhancing Rainfall Nowcasting Using Generative Deep Learning Model with Multi-Temporal Optical Flow

Precipitation nowcasting is critical for preventing damage to human life and the economy. Radar echo tracking methods such as optical flow algorithms have been widely employed for precipitation nowcasting because they can track precipitation motions well. Thus, this method, including the McGill algorithm for precipitation nowcasting by Lagrangian extrapolation (MAPLE), was implemented for operational precipitation nowcasting. However, advection-based methods struggle to predict the nonlinear motions of precipitation fields and dynamic processes, such as the growth and decay of precipitation. This study proposes an enhanced optical flow model using a multi-temporal optical flow field and a conditional generative adversarial network (cGAN). We trained the proposed model using a 3-year radar dataset provided by the Korean Meteorological Administration and performed forecast skill evaluations using both qualitative and quantitative methods. In particular, the model featuring multi-temporal optical flow enhances prediction accuracy for the nonlinear motion of precipitation fields, and the model’s accuracy can be further improved through the use of the cGAN structure. We have verified that these improvements hold for 0–3 h lead times. Based on this performance enhancement, we conclude that the multi-temporal optical flow model with cGAN has a potential role in operational precipitation nowcasting.

Optical measurements and flow modeling to characterize solid volume fractions in gravity-driven particle curtains

Solid volume fraction is a defining characteristic of multiphase gas–solid flows. In the case of gravity-driven particle curtains, it changes significantly along the flow direction. This has implications for the performance of falling-particle solar receivers, which use particle curtains to capture and store heat from concentrated sunlight. This study reports on the characterization of solid volume fraction as a function of height in gravity-driven particle curtains using optical techniques and simulations performed with a discrete element method (DEM) model. The experimental domain is a hopper-fed vertical channel, and the modeling domain is designed to match the experimental setup. Solid volume fraction was obtained from experimentally measured transmittance using two techniques: (1) a thermopile detector paired with a visible light source and (2) a high-speed camera. The DEM simulations were calibrated by identifying the coefficient of static friction between particles that resulted in equivalent mass flow rates to the experiments. All techniques, including an analytical model derived from first principles and empirical relationships, showed very similar trends of solid volume fraction decreasing along the flow direction, where it drops off rapidly initially before leveling off in a power law relationship. Both experimental and computational tests were performed with 1 mm and 2 mm particles, and solid volume fraction is found to decrease with increasing particle size at any location in the channel. Furthermore, the solid volume fractions calculated from transmittance measurements were found to be sensitive to the optical properties of the particles through Monte Carlo ray tracing simulations. To account for this effect, a new scale factor dependent on particle reflectivity is derived, which leads to significantly improved agreement between experimental and simulated results. A simple, closed-form equation is proposed to transform measured transmittance to a solid volume fraction while accounting for particle reflectivities. Even though solid volume fractions above 0.1 were not measured in this study, the technique developed is suitable to measure even larger solid volume fractions, especially for more reflective particles.

InvFlow: Involution and multi-scale interaction for unsupervised learning of optical flow

The convolution neural network is still the main tool for extracting the image features and the motion features for most of the optical flow models. The convolution neural networks cannot model the long-range dependencies, and more details are lost in deeper layers. All the deficiencies in the extracted features affect the estimated flow. Therefore, in this work, we concentrated on optimizing the convolution neural network in both the encoder and decoder parts to improve the image and motion features. To enhance the image features, we utilize the involution to provide rich features and model the long-range dependencies. In addition, we propose a Multi-Scale-Interaction module which utilizes the self-attention to make an interaction between the feature scales to avoid detail loss. Additionally, we propose a Motion-Features-Optimization block that utilizes the deformable convolution to enhance the motion features. Our model achieves the state-of-the-art performance on Sintel and KITTI 2015 benchmarks.

Stereo vision enabled flexible in-situ process authentication of additive manufacturing

Additive manufacturing (AM) has been increasingly adopted in aerospace, healthcare, and many other industries to fabricate parts with complex geometries. Despite the rapid market growth, the digital thread of AM makes it vulnerable to cyber-attacks, which raises concerns among researchers and practitioners. Notably, most of cyber-attacks maliciously manipulate the workpiece by altering the printing path. For AM authentication, it is important to monitor the printing process and detect malicious alterations in the printing path. Most existing studies mounted a camera onto the 3D printer and recorded videos of the extruder and printing bed to analyze the printing path. Besides the inconvenience and potential interferences to the AM process, these approaches are sensitive to camera positions and angles, and they are limited in the ability to precisely track the kinematics of the extruder and printing bed. To advance the state-of-the-art, this study leverages stereo vison and develops a new in-situ monitoring scheme for AM process authentication. With the stereo camera, RGB-D videos (RGB images + depth information) can be collected in real time from the extruder and printing bed. The kinematics of the extruder and bed are then estimated using optical flow and pinhole model for the reconstruction of the printing trajectory. Finally, a control chart is built upon the Hausdorff distance between the reconstructed path and the designed one (from the G-code) for the detection of printing path alterations. The developed scheme is evaluated and validated through case studies with multiple types of attack patterns.

ACCURATE MOTION ESTIMATION FOR ROTATIONAL IMAGE SEQUENCES

Complex motion patterns, non-rigid deformations, occlusions, and illumination changes largely affect rotational image sequences. This makes accurate motion estimation a challenging task. To address this issue, we propose an optical flow model to accurately estimate motion in rotational image sequences. Our model uses an additional constraint in the objective weighted with an edge-stopping function which allows non-zero curl specifically in the regions where rotation is involved. Our implementation of the model relies upon the robust Chambolle-Pock algorithm. We discuss the effects of the model parameters and the primal-dual parameters in the convergence of the algorithm. To further validate the effectiveness of our model, we combine our algorithm with some of the sophisticated implementation approaches with weighted median filtering. Our results on some of the rotational sequences from the Middlebury benchmark datasets show that our method achieves the best average angular and end-point errors when compared with some of the well-known models in the literature.

Cardinal viewpoints of 3D objects predicted by 2D optical flow model

Cardinal viewpoints of 3D objects predicted by 2D optical flow model

Physics-based optical flow estimation under varying illumination conditions

The physics-based optical flow (PBOF) model was derived from typical flow visualizations and it provided a more solid physical foundation for optical flow calculation. However, like many recent variational optical flow methods, it is not formulated to deal with the effect of illumination variation. In this paper, we propose a new efficient illumination-invariant optical flow estimation method, which improves the physics-based optical flow model. The proposed method introduces a multiplicative product of the velocity gradient and intensity from PBOF and an additive diffusion component to relax the brightness constancy assumption. The proposed method employs a linear transformation to reduce the influence of varying illumination on the data term. Then, a smoothness-sparsity regularization is used to constrain the minimization problem to have good edge-preserving for optical flow. Furthermore, the numerical implementation is given to solve the optimization problem. The accuracy of the developed method and the robustness against varying illumination are evaluated. The proposed method can provide motion estimation with acceptable accuracy for three optical flow datasets and cloud images of Jupiter’s great red spot, which have challenging illumination variations.

A method of motion estimation of segmental ventricular wall with tracking of ultrasonic echocardiogram

BackgroundUltrasonic echocardiography is commonly used for monitoring myocardial dysfunction. However, it has limitations such as poor quality of echocardiography images and subjective judgment of doctors.MethodsIn this paper, a calculation model based on optical flow tracking of echocardiogram is proposed for the quantitative estimation motion of the segmental wall. To improve the accuracy of optical flow estimation, a method based on confidence-optimized multiresolution(COM) optical flow model is proposed to reduce the estimation errors caused by the large amplitude of myocardial motion and the presence of “shadows” and other image quality problems. In addition, motion vector decomposition and dynamic tracking of the ventricular region of interest are used to extract information regarding the myocardial segmental motion. The proposed method was validated using simulation images and 50 clinical cases (25 patients and 25 healthy volunteers) for myocardial motion analysis.ResultsThe results demonstrated that the proposed method could track the motion information of myocardial segments well and reduce the estimation errors of optical flow caused due to the use of low-quality echocardiogram images.ConclusionsThe proposed method improves the accuracy of motion estimation for the cardiac ventricular wall.

An attention-mechanism incorporated deep recurrent optical flow network for particle image velocimetry

Particle image velocimetry (PIV), as a key technique in experimental fluid mechanics, is able to estimate complex velocity field through consecutive input particle images. In this study, an attention-mechanism incorporated deep recurrent network called ARaft-FlowNet has been proposed, on the basis of a previously established Recurrent All-Pairs Field Transforms optical flow model. The attention module is added to improve the network's capability of recognizing tracer particles' motion. Moreover, a parameterized dataset, ParaPIV-Dataset, is generated to explore the influence of particle parameters on deep learning networks, including particle diameter, image particle density, Gaussian noise, and peak intensity. The accuracy and generalizability of the newly proposed model has been evaluated and analyzed comprehensively. The results indicate that ARaft-FlowNet achieves state-of-the-art performance. Compared to previous methods, ARaft-FlowNet shows an accuracy improvement of 62.9%, 10.9%, and 9.4% in cylindrical flow, surface quasi-geostrophic flow, and DNS-turbulence flow. Meanwhile, the proposed model shows the strongest generalization and best capability to deal with complex flow fields with small-scale vortices. Additionally, tests on experimental turbulent jet data reveal that ARaft-FlowNet is able to deal with real PIV images with brightness variations and noise.

Unmanned Aerial Vehicle Position Estimation Augmentation Using Optical Flow Sensor

Flight control of unmanned aerial vehicle (UAV) requires reliable measurements of UAV’s position and attitude. Optical flow sensor is able to detect UAV’s motion with respect to the ground, and thus it can be incorporated in the integrated navigation system to enhance the positioning accuracy. However, the commonly used measurement model of optical flow sensor is actually fit for continuous-time condition only, since it defines the optical flow as the instantaneous velocity of a pixel on the image plane. For commercial optical flow sensors that work under discrete-time condition, a novel measurement model is proposed in this paper, which gives a vectorized symmetrical description of the optical flow measurement between every two successive frames in the image sequence. Moreover, a cubature transform-based data fusion scheme is also presented in this paper, which can directly augment the UAV’s position estimation with optical flow data, rather than extracting UAV’s velocity information from optical flow for dead-reckoning, and hence it can be easily added to the UAV’s navigation system without changing the existing algorithm flow. Flight tests are conducted using a quadcopter UAV that equipped with PIXHAWK autopilot and PX4Flow optical flow sensor, and the test results prove that the proposed optical flow model and data fusion scheme can effectively improve the accuracy of UAV position estimation in various outdoor environments.