Optical Flow-based Method Research Articles

Basketball technique action recognition using 3D convolutional neural networks

This research investigates the recognition of basketball techniques actions through the implementation of three-dimensional (3D) Convolutional Neural Networks (CNNs), aiming to enhance the accurate and automated identification of various actions in basketball games. Initially, basketball action sequences are extracted from publicly available basketball action datasets, followed by data preprocessing, including image sampling, data augmentation, and label processing. Subsequently, a novel action recognition model is proposed, combining 3D convolutions and Long Short-Term Memory (LSTM) networks to model temporal features and capture the spatiotemporal relationships and temporal information of actions. This facilitates the facilitating automatic learning of the spatiotemporal features associated with basketball actions. The model’s performance and robustness are further improved through the adoption of optimization algorithms, such as adaptive learning rate adjustment and regularization. The efficacy of the proposed method is verified through experiments conducted on three publicly available basketball action datasets: NTURGB + D, Basketball-Action-Dataset, and B3D Dataset. The results indicate that this approach achieves outstanding performance in basketball technique action recognition tasks across different datasets compared to two common traditional methods. Specifically, when compared to the frame difference-based method, this model exhibits a significant accuracy improvement of 15.1%. When compared to the optical flow-based method, this model demonstrates a substantial accuracy improvement of 12.4%. Moreover, this method showcases strong robustness, accurately recognizing actions under diverse lighting conditions and scenes, achieving an average accuracy of 93.1%. The research demonstrates that the method reported here effectively captures the spatiotemporal relationships of basketball actions, thereby providing reliable technical assessment tools for basketball coaches and players.

Manual Acupuncture Manipulation Recognition Method via Interactive Fusion of Spatial Multiscale Motion Features

Manual acupuncture manipulation (MAM) is essential in traditional Chinese medicine treatment. MAM action recognition is important for junior acupuncturists’ training and education; however, there are obvious personalized differences in hand gestures among expert acupuncturists for the same type of MAM. In addition, during the MAM operations, the magnitude and frequency of the expert acupuncturists’ hand shape and relative needle-holding finger position changes are tiny and fast, resulting in difficulties in observing MAM action details. Thus, we propose a Spatial Multiscale Interactive Fusion MAM Recognition Network to solve the difficulties in MAM recognition. First, this paper presents an optical flow-based hand motion contour global feature extraction method for acupuncture hand shape. Second, to explore the motion rule between the needle-holding fingers during the MAM operations, we design a quantitative description method of the relative motion of the needle-holding fingers: an “interactive attention module,” which achieves feature fusion and mines the correlation between different scales of MAM action features. Finally, the proposed MAM recognition method was validated by 20 acupuncturists from the Beijing University of Traditional Chinese Medicine and 10 from the Beijing Zhongguancun Hospital who participated in the MAM video signal collection. The proposed recognition method achieves the highest average validation accuracy of 95.3% and the highest test accuracy of 96.0% for four typical MAMs, proving its feasibility and effectiveness.

Experimental dataset investigation of deep recurrent optical flow learning for particle image velocimetry: flow past a circular cylinder

Current optical flow-based neural networks for particle image velocimetry (PIV) are largely trained on synthetic datasets emulating real-world scenarios. While synthetic datasets provide greater control and variation than what can be achieved using experimental datasets for supervised learning, it requires a deeper understanding of how or what factors dictate the learning behaviors of deep neural networks for PIV. In this study, we investigate the performance of the recurrent all-pairs field transforms-PIV (RAFTs-PIV) network, the current state-of-the-art deep learning architecture for PIV, by testing it on unseen experimentally generated datasets. The results from RAFT-PIV are compared with a conventional cross-correlation-based method, Adaptive PIV. The experimental PIV datasets were generated for a typical scenario of flow past a circular cylinder in a rectangular channel. These test datasets encompassed variations in particle diameters, particle seeding densities, and flow speeds, all falling within the parameter range used for training RAFT-PIV. We also explore how different image pre-processing techniques can impact and potentially enhance the performance of RAFT-PIV on real-world datasets. Thorough testing with real-world experimental PIV datasets reveals the resilience of the optical flow-based method’s variations to PIV hyperparameters, in contrast to the conventional PIV technique. The ensemble-averaged root mean squared errors between the RAFT-PIV and Adaptive PIV estimations generally range between 0.5–2 (px) and show a slight reduction as particle densities increase or Reynolds numbers decrease. Furthermore, findings indicate that employing image pre-processing techniques to enhance input particle image quality does not improve RAFT-PIV predictions; instead, it incurs higher computational costs and impacts estimations of small-scale structures.

Bi-directional Optical Flow-based Feature Point Tracking Method

In order to improve the robustness of the SLAM system based on the optical flow method, we propose a feature point tracking method with a bi-directional optical flow. The method first couples the forward LK optical flow and the reverse LK optical flow. Then, the feature points are screened and tracked by coupling the bidirectional optical flow. If the screened feature points do not reach the set threshold, they can be supplemented with either Fast points or Harris points. The experimental validation of the above method by VINS-Mono with the EuRoC dataset shows that VINS-Mono using this method significantly improves the accuracy of localization without increasing the running time cost compared with the original VINS-Mono.

An optical flow-based method for condition-based maintenance and operational safety in autonomous cleaning robots

Autonomous professional cleaning robots are ubiquitous today, coexist with humans, and the demand continues to improve hygiene and productivity, especially in large indoor workplaces. Hence, a proper maintenance strategy is essential to monitor the robot’s health, assuring flawless operation and safety. Manual supervision and periodic maintenance are adopted in general, which is challenging to detect failures in advance, resulting in higher maintenance costs and operational hazards. Anomalous vibration due to system degradation or environmental factors is an early symptom of failure or potential threats. Hence, predicting the sources of such abnormal vibration will help prompt maintenance action or set a hazard-free environment. However, such condition-monitoring research studies are not common for indoor cleaning robots. To fill this gap, we proposed an automated Condition Monitoring (CM) system, predicting the abnormal vibration sources that will enhance Condition-based Maintenance (CbM) and Operational Safety. A novel vibration-based CM method is presented for mobile robots, suitable for indoor environments under typical illumination states, using a monocular camera and optical flow technique instead of their usual perception-related applications. The sources of abnormal vibration were classified as terrain, collision, loose assembly, and structural imbalance. We modelled vibration as sparse optical flow, and the change in optical flow vector displacement due to the robot’s vibration is derived from consecutive camera frames used as vibration data. A practical framework is developed adopting the One Dimensional Convolutional Neural Network (1D CNN) model to fit this vibration data and tested predicting the vibration source class with an average accuracy of 93.8%. In addition, a vibration source map (CbM map) is proposed by fusing the predicted class in the workplace environment map for real-time monitoring. The case studies conducted using our in-house-developed cleaning robot show that the proposed CM framework will help the maintenance team for CbM, operational safety, and select proper maintenance strategies.

TempNet – temporal super-resolution of radar rainfall products with residual CNNs

AbstractThe temporal and spatial resolution of rainfall data is crucial for environmental modeling studies in which its variability in space and time is considered as a primary factor. Rainfall products from different remote sensing instruments (e.g., radar, satellite) have different space-time resolutions because of the differences in their sensing capabilities and post-processing methods. In this study, we developed a deep-learning approach that augments rainfall data with increased time resolutions to complement relatively lower-resolution products. We propose a neural network architecture based on Convolutional Neural Networks (CNNs), namely TempNet, to improve the temporal resolution of radar-based rainfall products and compare the proposed model with an optical flow-based interpolation method and CNN-baseline model. While TempNet achieves a mean absolute error of 0.332 mm/h, comparison methods achieve 0.35 and 0.341, respectively. The methodology presented in this study could be used for enhancing rainfall maps with better temporal resolution and imputation of missing frames in sequences of 2D rainfall maps to support hydrological and flood forecasting studies.

Contraction pressure analysis using optical imaging in normal and MYBPC3-mutated hiPSC-derived cardiomyocytes grown on matrices with tunable stiffness

Current in vivo disease models and analysis methods for cardiac drug development have been insufficient in providing accurate and reliable predictions of drug efficacy and safety. Here, we propose a custom optical flow-based analysis method to quantitatively measure recordings of contracting cardiomyocytes on polydimethylsiloxane (PDMS), compatible with medium-throughput systems.Movement of the PDMS was examined by covalently bound fluorescent beads on the PDMS surface, differences caused by increased substrate stiffness were compared, and cells were stimulated with β-agonist. We further validated the system using cardiomyocytes treated with endothelin-1 and compared their contractions against control and cells incubated with receptor antagonist bosentan. After validation we examined two MYBPC3-mutant patient-derived cell lines.Recordings showed that higher substrate stiffness resulted in higher contractile pressure, while beating frequency remained similar to control. β-agonist stimulation resulted in both higher beating frequency as well as higher pressure values during contraction and relaxation. Cells treated with endothelin-1 showed an increased beating frequency, but a lower contraction pressure. Cells treated with both endothelin-1 and bosentan remained at control level of beating frequency and pressure. Lastly, both MYBPC3-mutant lines showed a higher beating frequency and lower contraction pressure.Our validated method is capable of automatically quantifying contraction of hiPSC-derived cardiomyocytes on a PDMS substrate of known shear modulus, returning an absolute value. Our method could have major benefits in a medium-throughput setting.

Optical flow-based method to estimate internal wave parameters from X-band marine radar images

Optical flow-based method to estimate internal wave parameters from X-band marine radar images

Towards real-time tracking and counting of seedlings with a one-stage detector and optical flow

The population of crop seedlings is important for breeders and growers to evaluate the emergence rate of different cultivars and the necessity of replanting, but manual counting of plant seedlings is time-consuming and tedious. Building upon our prior work, we advanced the cotton seedling tracking method by incorporating a one-stage object detection deep neural network and optical flow to improve tracking speed and counting accuracy. Videos of cotton seedlings were captured using consumer-grade video cameras from the top view. You Only Look Once Version 4 (YOLOv4), a one-stage object detection network, was trained to detect cotton seedlings in each frame and to generate bounding boxes. To associate the same seedlings between adjacent frames, an optical flow-based tracking method was adopted to estimate camera motions. By comparing the positions of bounding boxes predicted by optical flow and detected by the YOLOv4 network in the same frame, the number of cotton seedlings was updated. The trained YOLOv4 model achieved high accuracy under conditions of occlusions, blurry images, complex backgrounds, and extreme illuminations. The F1 score of the final detection model was 0.98 and the average precision was 99.12%. Important tracking metrics were compared to evaluate the tracking performance. The Multiple-Object Tracking Accuracy (MOTA) and ID switch of the proposed tracking method were 72.8% and 0.1%, respectively. Counting results showed that the relative error of all testing videos was 3.13%. Compared with the Kalman filter and particle filter-based methods, our optical flow-based method generated fewer errors on testing videos because of higher accuracy of motion estimation. Compared with our previous work, the RMSE of the optical flow-based method decreased by 0.54 and the counting speed increased from 2.5 to 10.8 frames per second. The counting speed can reach 16.6 frames per second if the input resolution was reduced to 1280 × 720 pixels with an only 0.45% reduction in counting accuracy. The proposed method provides an automatic and near real-time tracking approach for counting of multiple cotton seedlings in video frames with improved speed and accuracy, which will benefit plant breeding and precision crop management.

Uncovering spatial representations from spatiotemporal patterns of rodent hippocampal field potentials

SummarySpatiotemporal patterns of large-scale spiking and field potentials of the rodent hippocampus encode spatial representations during maze runs, immobility, and sleep. Here, we show that multisite hippocampal field potential amplitude at ultra-high-frequency band (FPAuhf), a generalized form of multiunit activity, provides not only a fast and reliable reconstruction of the rodent's position when awake, but also a readout of replay content during sharp-wave ripples. This FPAuhf feature may serve as a robust real-time decoding strategy from large-scale recordings in closed-loop experiments. Furthermore, we develop unsupervised learning approaches to extract low-dimensional spatiotemporal FPAuhf features during run and ripple periods and to infer latent dynamical structures from lower-rank FPAuhf features. We also develop an optical flow-based method to identify propagating spatiotemporal LFP patterns from multisite array recordings, which can be used as a decoding application. Finally, we develop a prospective decoding strategy to predict an animal's future decision in goal-directed navigation.

Scene image and human skeleton-based dual-stream human action recognition

The dual stream-based human action recognition model offers the advantage of high recognition accuracy, but the algorithm is less robust in case of lighting changes. The human skeleton has a strong ability to express human behavior and actions; however, the scene information is ignored. Drawing on the idea of the dual-stream model, this paper proposes a human skeleton and scene image-based dual-stream model for human action recognition. The motion features are extracted through the spatio-temporal graph convolution of the human skeleton, and a scene recognition model is proposed based on the sparse frame sampling of video and video-level consensus strategy to process the scene video and gather the visual scene information. The proposed model exploits the advantages of skeleton information in motion expression and the superiority of the image in scene presentation. The scene information and spatio-temporal graph convolution-based human skeleton limbs are fused complementarily to achieve human action recognition. Compared to the conventional optical flow-based dual-stream action recognition method, this model is verified by experimenting under unstable light conditions, and the performance of human action recognition is robust and promising.

FIB/SEM tomography segmentation by optical flow estimation

Focused ion beam/scanning electron microscopy tomography (FIB/SEM tomography) is the method of choice for the tomographic reconstruction of mesoporous materials systems in various fields such as batteries, fuel cells, filter applications or composite materials. However, due to so called shine-through artifacts in FIB/SEM tomographies of porous materials, their segmentation into pore space and solid material is a nontrivial task. Here, an optical flow-based method that utilizes shine-through artifacts for segmentation is introduced. Subsequently, the performance of the method is discussed by means of tomographic datasets of a polymer electrolyte fuel cell catalyst layer and a lithium ion battery composite electrode. Previous, manual segmentations of these datasets allow the evaluation of the results – for the catalyst layer an accuracy of 86.6% and a precision of 84.0% is reached. In both cases, the optical flow-based approach gives significantly better results than comparable segmentations obtained from gray-value threshold binarization.

Biprediction-Based Video Quality Enhancement via Learning.

Convolutional neural networks (CNNs)-based video quality enhancement generally employs optical flow for pixelwise motion estimation and compensation, followed by utilizing motion-compensated frames and jointly exploring the spatiotemporal correlation across frames to facilitate the enhancement. This method, called the optical-flow-based method (OPT), usually achieves high accuracy at the expense of high computational complexity. In this article, we develop a new framework, referred to as biprediction-based multiframe video enhancement (PMVE), to achieve a one-pass enhancement procedure. PMVE designs two networks, that is, the prediction network (Pred-net) and the frame-fusion network (FF-net), to implement the two steps of synthesization and fusion, respectively. Specifically, the Pred-net leverages frame pairs to synthesize the so-called virtual frames (VFs) for those low-quality frames (LFs) through biprediction. Afterward, the slowly fused FF-net takes the VFs as the input to extract the correlation across the VFs and the related LFs, to obtain an enhanced version of those LFs. Such a framework allows PMVE to leverage the cross-correlation between successive frames for enhancement, hence capable of achieving high accuracy performance. Meanwhile, PMVE effectively avoids the explicit operations of motion estimation and compensation, hence greatly reducing the complexity compared to OPT. The experimental results demonstrate that the peak signal-to-noise ratio (PSNR) performance of PMVE is fully on par with that of OPT while its computational complexity is only 1% of OPT. Compared with other state-of-the-art methods in the literature, PMVE is also confirmed to achieve superior performance in both objective quality and visual quality at a reasonable complexity level. For instance, PMVE can surpass its best counterpart method by up to 0.42 dB in PSNR.

Dynamic tongue area measurements in ultrasound images for adults with obstructive sleep apnea.

Obstructive sleep apnea is a chronic breathing disorder where the recursive collapse of the upper airway causes cessation of airflow during sleep. Quantitative assessments of dynamic tongue motion may provide a better understanding of the mechanism of obstructive sleep apnea. Tongue area changes, in submental ultrasound images recorded during wakefulness from normal breathing by the Müller manoeuvre, were tracked using the modified optical flow-based method. The results demonstrated that patients with obstructive sleep apnea had a larger mid-sagittal tongue area compared to the control group (during normal breathing, p=.004, during the Müller manoeuvre p=.005, and differences between normal breathing and the Müller manoeuvre, p=.01). Tongue area measurements were significantly different in patients with varying severity of OSA during normal breathing and during the Müller manoeuvre, and had differences between normal breathing and the Müller manoeuvre (p=.005, p=.008 and p=.03, respectively). Patients with moderate or severe obstructive sleep apnea exhibited minimal movement of the tongue. In contrast, normal controls and patients with mild obstructive sleep apnea demonstrated bidirectional tongue motions during a transition from normal breathing to the Müller manoeuvre in wakefulness. Identifying different patterns of deformation and displacement of the tongue may have potential in evaluating the presence and the pathogenesis of OSA.

Respiration Monitoring for Premature Neonates in NICU

In this paper, we investigate an automated pipeline to estimate respiration signals from videos for premature infants in neonatal intensive care units (NICUs). Two flow estimation methods, namely the conventional optical flow- and deep learning-based flow estimation methods, were employed and compared to estimate pixel motion vectors between adjacent video frames. The respiratory signal is further extracted via motion factorization. The proposed methods were evaluated by comparing our automated extracted respiration signals to that extracted from chest impedance on videos of five premature infants. The overall average cross-correlation coefficients are 0.70 for the optical flow-based method and 0.74 for the deep flow-based method. The average root mean-squared errors are 6.10 and 4.55 for the optical flow- and the deep flow-based methods, respectively. The experimental results are promising for further investigation and clinical application of the video-based respiration monitoring method for infants in NICUs.

Detection of motion-compensated frame-rate up-conversion via optical flow-based prediction residue

To increase the temporal continuity of low frame-rate videos, Motion-Compensated Frame-Rate Up-Conversion (MC-FRUC), which is a special frame based video editing manipulation, can be employed to synthesize new intermediate frame between two successive frames. However, MC-FRUC can also be utilized to fake high frame-rate videos causing the wastes of storage space or users misleading. Although existing MC-FRUC detectors can obtain desirable performance for uncompressed videos or compressed videos with high perceptual quality, theirs performance degrades when test videos are encoded with higher quantization parameters. To deal with this problem, this paper presents a new optical flow-based detection method. Our method is divided into four steps. First, the forward or backward optical flow map is estimated by optical flow algorithm between the candidate frame, and its previous frame or following frame, respectively. Second, the prediction residue is calculated on the basis of these estimated optical flow maps to eliminate the side effect of texture information in the non-motion regions and the motion regions. Third, Markov statistic features are extracted. Finally, these features are fed into the ensemble classifier for identification of interpolated frames. We test different types of MC-FRUC methods under higher quantization parameters, and the results have shown the effectiveness of the proposed method compared with the existing relative ones.

Virtual reality for animal navigation with camera-based optical flow tracking

BackgroundVirtual reality combined with a spherical treadmill is used across species for studying neural circuits underlying navigation and learning. New methodWe developed an optical flow-based method for tracking treadmill ball motion in real time using a single high-resolution camera. ResultsTracking accuracy and timing were determined using calibration data. Ball tracking was performed at 500 Hz and integrated with an open source game engine for virtual reality projection. The projection was updated at 120 Hz with a latency with respect to ball motion of 30 ± 8 ms. The system was tested for behavior with fruit flies. The application and source code are available at https://github.com/ivan-vishniakou/neural-circuits-vr. Comparison with existing method(s)Optical flow-based tracking of treadmill motion is typically achieved using optical mice. The camera-based optical flow tracking system developed here is based on off-the-shelf components and offers control over the image acquisition and processing parameters. This results in flexibility with respect to tracking conditions – such as ball surface texture, lighting conditions, or ball size – as well as camera alignment and calibration. ConclusionsA fast system for rotational ball motion tracking suitable for virtual reality behavior with fruit flies was developed and characterized.

Projection Distortion-based Object Tracking in Shader Lamp Scenarios.

Shader lamp systems augment the real environment by projecting new textures on known target geometries. In dynamic scenes, object tracking maintains the illusion if the physical and virtual objects are well aligned. However, traditional trackers based on texture or contour information are often distracted by the projected content and tend to fail. In this paper, we present a model-based tracking strategy, which directly takes advantage from the projected content for pose estimation in a projector-camera system. An iterative pose estimation algorithm captures and exploits visible distortions caused by object movements. In a closed-loop, the corrected pose allows the update of the projection for the subsequent frame. Synthetic frames simulating the projection on the model are rendered and an optical flow-based method minimizes the difference between edges of the rendered and the camera image. Since the thresholds automatically adapt to the synthetic image, a complicated radiometric calibration can be avoided. The pixel-wise linear optimization is designed to be easily implemented on the GPU. Our approach can be combined with a regular contour-based tracker and is transferable to other problems, like the estimation of the extrinsic pose between projector and camera. We evaluate our procedure with real and synthetic images and obtain very precise registration results.

Spatial Multilevel Optical Flow Architecture-based Dynamic Motion Estimation in Vehicular Traffic Scenarios

Pedestrian detection is a challenging area in the intelligent vehicles domain. During the last years, many works have been proposed to efficiently detect motion in images. However, the problem becomes more complex when it comes to detecting moving areas while the vehicle is also moving. This paper presents a variational optical flow-based method for motion estimation in vehicular traffic scenarios. We introduce a framework for detecting motion areas with small and large displacements by computing optical flow using a multilevel architecture. The flow field is estimated at the shortest level and then successively computed until the largest level. We include a filtering parameter and a warping process using bicubic interpolation to combine the intermediate flow fields computed at each level during optimization to gain better performance. Furthermore, we find that by including a penalization function, our system is able to effectively reduce the presence of outliers and deal with all expected circumstances in real scenes. Experimental results are performed on various image sequences from Daimler Pedestrian Dataset that includes urban traffic scenarios. Our evaluation demonstrates that despite the complexity of the evaluated scenes, the motion areas with both moving and static camera can be effectively identified.

Stable stitching method for stereoscopic panoramic video

This paper proposes a stable method of generating a stereoscopic panoramic video with the omnidirectional stereo (ODS) format. Different from the traditional image stitching method which can only be applied to generate a monocular panorama, we adopt an optical flow-based blending method to create two panoramas for binocular vision. In addition, traditional image stitching methods based on seam-finding tend to cause the problem of temporal flicker. We address this problem by restricting the optical flow field of the new frame with its previous frame's optical flow field. Thus, the generated video is stable and out of temporal flicker. There are four key operations in our approach. First, we adopt the ODS format which is the basis of stereoscopic panorama. Second, we do effective exposure compensation, making the brightness of the two eyes' panoramas consistent. Third, we employ an optical flow-based blending method to synthesis the final panorama effectively. Fourth, we take the previous frame's optical flow field as the restriction of the present frame's optical flow field to acquire a stable video. The final output videos can deliver a pleasant and impressive stereoscopic viewing experience to the audience when the audience watches the videos in the virtual reality headset.