Optical Flow Method Research Articles

Deepfake Video Detection Using Facial Feature Points and Ch-Transformer

With the development of Metaverse technology, the avatar in Metaverse has faced serious security and privacy concerns. Analyzing facial features to distinguish between genuine and manipulated facial videos holds significant research importance for ensuring the authenticity of characters in the virtual world and for mitigating discrimination, as well as preventing malicious use of facial data. To address this issue, the Facial Feature Points and Ch-Transformer (FFP-ChT) deepfake video detection model is designed based on the clues of different facial feature points distribution in real and fake videos and different displacement distances of real and fake facial feature points between frames. The face video input is first detected by the BlazeFace model, and the face detection results are fed into the FaceMesh model to extract 468 facial feature points. Then the Lucas-Kanade (LK) optical flow method is used to track the points of the face, the face calibration algorithm is introduced to re-calibrate the facial feature points, and the jitter displacement is calculated by tracking the facial feature points between frames. Finally, the Class-head (Ch) is designed in the transformer, and the facial feature points and facial feature point displacement are jointly classified through the Ch-Transformer model. In this way, the designed Ch-Transformer classifier is able to accurately and effectively identify deepfake videos. Experiments on open datasets clearly demonstrate the effectiveness and generalization capabilities of our approach.

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

Sign language (SL) is vital in fostering communication for the deaf and hard-of-hearing communities. Continuous Sign Language Translation (CSLT) is a work that translates sign language into spoken language. CSLT translation is done by changing continuous forms into isolated signs. Segmenting morpheme signs from phrase signs has several challenges, such as the availability of annotated datasets and the complexity of continuous gesture movements. The Indonesian Sign Language (SIBI) system follows Indonesian grammatical norms, including word formation, in contrast to other sign languages with rules derived from their spoken language. In SIBI, a word can consist of a root word and an affix word. Therefore, temporal action segmentation in SIBI is important to reconstruct the results of translating each sign into spoken Indonesian sentences. This research uses an optical flow approach to segment temporal actions in SIBI videos. Optical flow methods that calculate changes in intensity between adjacent frames can be used to determine the occurrence of sign movement or vice versa to determine the delay between sign movements. The absence of intensity differences between the two frames indicates the boundary between sign gestures. This study tested the use of dense optical flow on videos containing SIBI sentences taken from 3 signers. Evaluation is done on several parameters in the dense optical flow algorithm, such as threshold size, PyrScale, and WinSize, to obtain the best accuracy. This paper shows that the optical flow algorithm successfully performs segmentation, as measured by Perf and F1r. The experimental results showed that the highest Perf and F1r yields were 0.8298 and 0.8524, respectively.

A simultaneous non-invasive diagnosis method for flame temperature and velocity fields based on radiation self-calibration and optical flow

The measurement of flame temperature and velocity is crucial to combustion research, hence it is of great significance to develop a non-invasive diagnosis technology for the measurement of flame temperature and velocity. In this study, a single-camera simultaneous non-invasive diagnosis technology for flame temperature and velocity field was developed, based on radiation self-calibration and optical flow method. The diagnosis method was applied to two case studies, propane diffusion flame and solid surface spread flame. The reconstructed flame temperature field was compared with measuring results by thermocouples. Results show that the reconstructed temperature field can accurately describe the symmetrical distribution characteristics and the clear high-temperature front. The combustion zone reconstruction temperature was 1250–1380 K for propane diffusion flame and the maximum reconstruction temperature was 1400 K for solid spread flame. The maximum relative error between the reconstructed temperature and the measured values by thermocouples is less than 15 %. Additionally, the flow velocity field at the flame edge was also measured, which could provide a good qualitative characterization of flame kinematic.

Gradient-Based Optimization Method for Experimental Modal Parameter Estimation with Finite Element Model

This paper presents a novel gradient-based optimization algorithm for improving the accuracy of experimentally estimated modal parameters with the assistance of finite element models. Initially, we recast the discrete vibration response equation into a matrix form and formulate the parameter estimation problem in modal analysis as an optimization problem. Then the problem is solved with a gradient-based iterative algorithm, which explicitly exhibits the closed form of gradients used in optimization. Initial values for this iteration are parameters derived from finite element models, since every important engineering structure should be analyzed with a finite element model before it is constructed. Subsequently, the performance of this algorithm is validated by both pure numerical experiments, which simulate the physical world, and experiments using real measurement data gathered by sensors in the real physical world. The algorithm’s performance is further enhanced by incorporating gradient clipping and an adaptive iteration threshold. As a comparison, a discussion on classical least-squares time-domain method for the problem is provided. For practical applications, the Shi–Tomasi corner detection and Lucas–Kanade optical flow methods are deployed to detect corner points from videos taken during the vibration of a structure and track the motion of these points in the videos.

A Dynamic Visual SLAM System Incorporating Object Tracking for UAVs

The capability of unmanned aerial vehicles (UAVs) to capture and utilize dynamic object information assumes critical significance for decision making and scene understanding. This paper presents a method for UAV relative positioning and target tracking based on a visual simultaneousocalization and mapping (SLAM) framework. By integrating an object detection neural network into the SLAM framework, this method can detect moving objects and effectively reconstruct the 3D map of the environment from image sequences. For multiple object tracking tasks, we combine the region matching of semantic detection boxes and the point matching of the optical flow method to perform dynamic object association. This joint association strategy can prevent trackingoss due to the small proportion of the object in the whole image sequence. To address the problem ofacking scale information in the visual SLAM system, we recover the altitude data based on a RANSAC-based plane estimation approach. The proposed method is tested on both the self-created UAV dataset and the KITTI dataset to evaluate its performance. The results demonstrate the robustness and effectiveness of the solution in facilitating UAV flights.

FFD-SLAM: A Real-Time Visual SLAM Toward Dynamic Scenes with Semantic and Optical Flow Information

To solve the problem of poor localization accuracy and robustness of visual simultaneous localization and mapping (SLAM) systems in highly dynamic environments, this paper proposes a dynamic visual SLAM algorithm called FFD-SLAM that fuses the target detection network with the optical flow method. The algorithm considers ORB-SLAM2 as the basic framework, joins the semantic thread in parallel with its tracking thread, initially obtains the set of feature points through the real-time detection of dynamic objects in the environment through YOLOv5 in the semantic thread, then filters the set of feature points obtained in the semantic thread through the optical flow module, and finally utilizes the remaining static feature points for the matching calculation. Experiments showed that the proposed algorithm showed an improvement of approximately 97% in the localization accuracy compared with the ORB-SLAM2 algorithm in a highly dynamic environment, which effectively improves the localization accuracy and robustness of the system. The proposed algorithm also showed a higher real-time performance compared with some excellent dynamic SLAM algorithms.

A noise-robust vibration signal extraction method utilizing intensity optical flow

A noise-robust intensity optical flow (IOF) method was developed by leveraging the linear relationship between the vibrating displacement and optical intensity variation. By identifying measurement points with signal variance of regions of interest, this method fuses all the signals from these points into one integrated signal, and then decomposes it into various vibration mode signals and noise components. As a result, the vibration signals can be obtained with significant improvement of signal-to-noise ratio. Compared with existing advanced methods, our method is straightforward to perform better measured accuracy but preserve computationally efficient, even at the kilohertz level. Simulations and experiments are demonstrated to verify the capability and accuracy of this simple but effective method. The results show that the proposed method yields a correlation coefficient of 99.75 % with the identification results and a speed increase of > 30 % in contrast to the phase optical flow method.

Space-Charge Effects on Transverse Emittance in the Extraction Region of the RIKEN 28-GHz ECRIS

To confirm the significance of space-charge effects (SCEs) on the emittance growth in the “extraction region” of the RIKEN 28-GHz electron-cyclotron-resonance ion source (ECRIS), we reconstructed the phase-space distribution of argon (Ar) beams traveling through high-intensity nitrogen (N) beams and reaching at the end of “extraction region”. Here, the “extraction region” is defined as the low-energy beam transport region between the ECRIS puller electrode and a magnetic analyzer. The transverse phase-space distribution (PSD) of the Ar beams, which were exposed to the SCEs due to the high-intensity N beams, was measured after magnetic analysis using the pepper-pot emittance meter (PPEM). Because we suppressed each Ar beam current below 50 µA, the expansion in the transverse phase-space distribution due to the SCEs after the magnetic analyzer could be sufficiently reduced. Sufficient angular resolution was achieved by improving the image analysis of PPEM measurements with the “optical flow method”. Therefore, we obtained a transverse PSD at the endpoint of the extraction region of the ECRIS using the three dimensional backtracking. The obtained rms emittance of the Ar10,11,13+ were found to increas as a function of the extraction current I ext, while those of Ar16+ were found to be nearly constant. The reconstructed PSD at the end of the “extraction region” shows that the smaller the M/Q, in other words, the larger the charge state Q, the smaller both the positional and angular distribution for each I ext. To reveal the space charge effect on the beam trajectory passing though the “extraction region”, further experimental measurements and analyses are underway, including detailed trajectory simulation.

Impacts of flash boiling plume interactions on atomization evaluated using two-dimensional optical slit nozzles

Flash boiling atomization is a promising approach for liquid fuel energy conversion purposes, especially for alternative fuels and zero-carbon fuels. However, the fundamental mechanisms of flash boiling atomization is quite different from typical sub-cooled atomization both in the primary breakup mechanisms and plume interference principles. This work aims to investigate the spray plume interaction impacts under flash boiling injection schemes. To reduce the complexity in handling the dense spray regime for practical spray injectors, two-dimensional slit nozzles created by quartz blocks and thin channels are employed. Divergent, straight, and converge nozzles are designed and examined to evaluate the impact of nozzle design under the same thermodynamic conditions. Nozzles with a 2D expansion cavity are also tested to further reduce the optical depth in the observation direction. Various optical diagnostics methods including backlit measurement, phase Doppler interferometry, and the optical flow method are used to investigate different characteristics of the resultant superheat flows from different perspective. The experimental investigations reveal the interference mechanism of the two-plume nozzle with the increment of fuel temperature, as well as its impacts on atomization efficiency from the perspective of droplet sizes.

Gas–Liquid Two-Phase Flow Measurement Based on Optical Flow Method with Machine Learning Optimization Model

Gas–Liquid two-phase flows are a common flow in industrial production processes. Since these flows inherently consist of discrete phases, it is challenging to accurately measure the flow parameters. In this context, a novel approach is proposed that combines the pyramidal Lucas-Kanade (L–K) optical flow method with the Split Comparison (SC) model measurement method. In the proposed approach, videos of gas–liquid two-phase flows are captured using a camera, and optical flow data are acquired from the flow videos using the pyramid L–K optical flow detection method. To address the issue of data clutter in optical flow extraction, a dynamic median value screening method is introduced to optimize the corner point for optical flow calculations. Machine learning algorithms are employed for the prediction model, yielding high flow prediction accuracy in experimental tests. Results demonstrate that the gradient boosted regression (GBR) model is the most effective among the five preset models, and the optimized SC model significantly improves measurement accuracy compared to the GBR model, achieving an R2 value of 0.97, RMSE of 0.74 m3/h, MAE of 0.52 m3/h, and MAPE of 8.0%. This method offers a new approach for monitoring flows in industrial production processes such as oil and gas.

A gated recurrent unit model based on ultrasound images of dynamic tongue movement for determining the severity of obstructive sleep apnea

Obstructive sleep apnea (OSA) presents as a respiratory disorder characterized by recurrent upper pharyngeal airway collapse during sleep. Dynamic tongue movement (DTM) analysis emerges as a promising avenue for elucidating the pathophysiological underpinnings of OSA, thereby facilitating its diagnosis. Recent endeavors have utilized artificial intelligence techniques to categorize OSA severity leveraging electrocardiography and blood oxygen saturation data. Nonetheless, the integration of ultrasound (US) imaging of the tongue remains largely untapped in the development of machine learning models aimed at determining the severity of OSA. This study endeavors to bridge this gap by capturing US images of DTM dynamics during wakefulness, encompassing transitions from normal breathing (NB) to the performance of the Müller maneuver (MM) in a cohort of 53 patients. Leveraging the modified optical flow method (MOFM), the trajectories of patients’ DTM were tracked, facililtating the extraction of 27 parameters vital for model training. These parameters encompassed nine-point lateral movement, nine-point axial movement, and nine-point total displacement of the tongue, resulting in a dataset of 186,030 samples. The gated recurrent unit (GRU) method, renowned for its efficacy in motion tracking, was employed for model development in this study. Validation of the developed model was conducted via stratified k-fold cross-validation (SCV). The systems’ overall performance in classifying OSA severity, as quantified by mean accuracy (MA), yielded a value of 43.49%. This pilot investigation marks an exploratory endeavor into the utilization of artificial intelligence for the classification of OSA severity based on US images and dynamic movement patterns. This novel model holds potential to assist clinicians in categorizing OSA severity and guiding the selection of pertinent treatment modalities tailored to the individual needs of patients afflicted with OSA.

Research on variational optical flow method for rockfall monitoring

Research on variational optical flow method for rockfall monitoring

First In-Vivo demonstration of 1000fps High Speed Coronary Angiography (HSCA) in a swine animal model.

High-speed-angiography (HSA) 1000 fps imaging was successfully used previously to visualize contrast media/blood flow in neurovascular anatomies. In this work we explore its usage in cardiovascular anatomies in a swine animal model. A 5 French catheter was guided into the right coronary artery of a swine, followed by the injection of iodine contrast through a computer-controlled injector at a controlled rate of 40 (ml/min). The injection process was captured using high-speed angiography at a rate of 1000 fps. The noise in the images was reduced using a custom built machine-learning model consisting of Long Short-term memory networks. From the noise reduced images, velocity profiles of contrast/blood flow through the artery was calculated using Horn-Schunck optical flow (OF) method. From the high-speed coronary angiography (HSCA) images, the bolus of contrast could be visually tracked with ease as it traversed from the catheter tip through the artery. The imaging technique's high temporal resolution effectively minimized motion artifacts resulting from the heart's activity. The OF results of the contrast injection show velocities in the artery ranging from 20 - 40 cm/s. The results demonstrate the potential of 1000 fps HSCA in cardiovascular imaging. The combined high spatial and temporal resolution offered by this technique allows for the derivation of velocity profiles throughout the artery's structure, including regions distal and proximal to stenoses. This information can potentially be used to determine the need for stenoses treatment. Further investigations are warranted to expand our understanding of the applications of HSCA in cardiovascular research and clinical practice.

Demonstration of 1000 fps High-Speed Angiography (HSA) in Pre-Clinical In-vivo Rabbit Aneurysm Models During Flow-Diverter Treatment.

High Speed Angiography (HSA) at 1000 fps is a novel interventional-imaging technique that was previously used to visualize changes in vascular flow details before and after flow-diverter treatment of cerebral aneurysms in in-vitro 3D printed models.1 In this first pre-clinical work, we demonstrate the use of the HSA technique during flow-diverter treatment of in-vivo rabbit aneurysm models. An aneurysm was created in the right common carotid artery of each of two rabbits using previously published elastase aneurysm-creation methods.2 A 5 French catheter was inserted into the femoral artery and moved to the aneurysm location under the guidance of standard-speed 10 fps Flat Panel Detector (FPD) fluoroscopy. Following this, a flow diverter stent was placed in the parent vessel covering the aneurysm neck and diverting the flow away from the aneurysm. HSA was performed before and after placement of the flow diverter using a 1000 fps CdTe photon-counting detector (Aries, Varex). The detector was mounted on a motorized changer and was used with a commercial x-ray c-arm system (Fig. 1). During these procedures Omnipaque iodinated contrast was injected into the aneurysm area using a computer-controlled injector at a steady rate of 50 ml/min or 70 ml/min depending on the rabbit to visualize blood flow detail. The contrast injection and x-ray image acquisition were synchronized manually. The x-ray image acquisition was for a duration of 1 second, from which 300 ms was used for velocity analysis during systole. Detailed differences in flow patterns in the region of interest (ROI) between pre and post flow-diverter deployment were visualized at the high frame rates. The Optical Flow (OF) method for velocity calculation was performed upon the acquired 1000 fps HSA image sequences to provide quantitative evaluation of flow.

A Hybrid PIV/Optical Flow Method for Incompressible Turbulent Flows

We present novel velocimetry algorithms based on the hybridization of correlation-based Particle Image Velocimetry (PIV) and a combination of Lucas–Kanade and Liu–Shen optical flow (OpF) methods. An efficient Aparapi/OpenCL implementation of those methods is also provided in the accompanying open-source QuickLabPIV-ng tool enabled with a Graphical User Interface (GUI). Two different options of hybridization were developed and tested: OpF as a last step, after correlation-based PIV, and OpF as a substitute for sub-pixel interpolation. Hybridization increases the spatial resolution of PIV, enabling the characterization of small turbulent scales and the computation of key turbulence parameters such as the rate of dissipation of turbulent kinetic energy. The method was evaluated using both synthetic and real databases, representing flows that exhibit a variety of locally isotropic homogeneous turbulent scales. The proposed hybrid PIV-OpF results in a 3-fold increase in the PIV density for synthetic images. The analysis of power spectral density functions and auto-correlation demonstrated the impact of PIV image quality on the accuracy of the method and its ability to extend the turbulence range. We discuss the challenges posed by optical noise and tracer density in the quality of the vector map density.

Recurrent Partial Kernel Network for Efficient Optical Flow Estimation

Optical flow estimation is a challenging task consisting of predicting per-pixel motion vectors between images. Recent methods have employed larger and more complex models to improve the estimation accuracy. However, this impacts the widespread adoption of optical flow methods and makes it harder to train more general models since the optical flow data is hard to obtain. This paper proposes a small and efficient model for optical flow estimation. We design a new spatial recurrent encoder that extracts discriminative features at a significantly reduced size. Unlike standard recurrent units, we utilize Partial Kernel Convolution (PKConv) layers to produce variable multi-scale features with a single shared block. We also design efficient Separable Large Kernels (SLK) to capture large context information with low computational cost. Experiments on public benchmarks show that we achieve state-of-the-art generalization performance while requiring significantly fewer parameters and memory than competing methods. Our model ranks first in the Spring benchmark without finetuning, improving the results by over 10% while requiring an order of magnitude fewer FLOPs and over four times less memory than the following published method without finetuning. The code is available at github.com/hmorimitsu/ptlflow/tree/main/ptlflow/models/rpknet.

Improving Audio-Visual Segmentation with Bidirectional Generation

The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. Code is released in: https://github.com/OpenNLPLab/AVS-bidirectional.

Mode-shape magnification in high-speed camera measurements

If motion, identified using image-based methods, is too small to be seen with the naked eye, motion magnification can be used to help with the visualization. Established motion-magnification methods magnify (typically up to 1000 times) the band-passed content of the image data. Especially at higher frequencies, the amplitudes of measured displacements are often below the noise floor. In this research, a novel method for amplifying vibrations, based on experimental modal analysis (EMA), is introduced. The response of the examined structure to dynamic excitation is measured with a simplified, gradient-based, optical flow method and used to perform a hybrid modal analysis in conjunction with a reference accelerometer response measurement. Such a hybrid approach can: (a) identify structural dynamics significantly in the sub-pixel range, and (b) significantly below the image noise floor. The image of the vibrating structure is subdivided using a planar triangle mesh, which is then warped in accordance with the identified mode shape. A mesh-element-wise affine transformation is performed to obtain an image of the magnified mode shape. In the experimental part, the proposed method achieved magnification factors of approximately 40 thousand times, which is an order of magnitude deeper into noise than available before; additionally, the proposed approach is numerically significantly less demanding.The introduced mode-shape magnification presents an alternative to existing motion-magnification methods for applications where the harmonic displacement information is hidden by image noise.

Real-time tracking of radial artery vessel wall using a Kalman filter-based ultrasound single-plane wave RF signal time-frequency information fusion algorithm

ObjectiveThe goal of our study was to achieve real-time dynamic tracking of the radial artery vessel wall in everyday life, which is crucial for blood pressure estimation and cardiovascular disease prediction. MethodsThe algorithm integrates time–frequency information from ultrasound single-plane wave RF (radiofrequency) signals and consists of three main components: feature frame ROI (region of interest) selection, inter-frame cross-correlation and intra-frame autocorrelation, and feature frame Kalman filtering. ResultsExperimental validation on a silicone gel ultrasound phantom with simulated blood vessels demonstrates an average diameter estimation error of less than 0.5% compared to ground truth values. Comparative experiments show similar relative errors (2.83%, 2.43%) to the optical flow method, with a computational speed 7.32 times faster. The algorithm accurately aligns extracted pulse waves with pressure pulse waves at six local feature points per cycle. ConclusionThe proposed method achieves a favorable balance between speed and accuracy in tracking the radial artery vessel wall. It holds significant potential for real-time monitoring of physiological health parameters, offering precise and dynamic tracking capabilities. SignificanceThis algorithm addresses the challenges of non-invasive real-time tracking, benefiting blood pressure estimation and cardiovascular disease prediction. Its integration of time–frequency information and efficient computational speed make it valuable for the scientific community and public in monitoring physiological health parameters.

A Crowd Movement Analysis Method Based on Radar Particle Flow.

Crowd movement analysis (CMA) is a key technology in the field of public safety. This technology provides reference for identifying potential hazards in public places by analyzing crowd aggregation and dispersion behavior. Traditional video processing techniques are susceptible to factors such as environmental lighting and depth of field when analyzing crowd movements, so cannot accurately locate the source of events. Radar, on the other hand, offers all-weather distance and angle measurements, effectively compensating for the shortcomings of video surveillance. This paper proposes a crowd motion analysis method based on radar particle flow (RPF). Firstly, radar particle flow is extracted from adjacent frames of millimeter-wave radar point sets by utilizing the optical flow method. Then, a new concept of micro-source is defined to describe whether any two RPF vectors originated from or reach the same location. Finally, in each local area, the internal micro-sources are counted to form a local diffusion potential, which characterizes the movement state of the crowd. The proposed algorithm is validated in real scenarios. By analyzing and processing radar data on aggregation, dispersion, and normal movements, the algorithm is able to effectively identify these movements with an accuracy rate of no less than 88%.