Phase-based Optical Flow Research Articles

3D Vibration Analysis using Phase-Based Displacement Estimation and Stereo Cameras

Non-contact sensors are increasingly integral to the field of Structural Health Monitoring (SHM), with camera being widely utilized due to their high spatial resolution and cost-effectiveness in measuring structural displacements. Phase-based optical flow (POF) techniques have proven valuable for accurate and computationally efficient 2D vibration measurements. However, these techniques are typically confined to single-camera systems and encounter challenges in measuring vibrations in out-of-plane directions. Previous research on 3D vibration measurement has predominantly centered on pixel intensity-based techniques using stereo camera systems. Although these methods demonstrate high performance in calculating global strains and stresses, they are less suitable for accurate vibration measurement. To address these limitations, this study proposes a novel 3D vibration measurement approach that integrates POF within the context of stereo camera systems. 2D vibrations are extracted by applying Complex Gabor filters to stereo images, and subsequently, 3D vibration measurements are estimated through camera calibration and triangulation, using computed camera internal and external parameters. To validate the method, modal analysis experiments are conducted on an aluminum plate. The 3D vibration measurement results are compared with those obtained through 3D Digital Image Correlation (DIC) and contact sensors and the accuracy is demonstrated.

Phase-based Full-field Displacement Measurement with Phase Nonlinearity Weighting Process

The camera, as a non-contact sensor, has evolved into a robust tool for measuring the full-field vibration of complex structures. It offers distinct advantages over traditional contact sensors, including flexible positioning, simultaneous multi-point tracking, and high spatial resolution. Vibration imaging techniques have been developed to identify structural operational conditions through feature extraction methods. Phase-based vibration imaging techniques can visualize full-field operational shapes or vibrational features in a less-supervised manner. While phase-based optical flow with Gabor filters provides accurate displacement estimation, managing a large amount of image data can pose challenges in extracting full-field displacement efficiently. In this paper, we propose a compressive sensing approach for full-field phase-based optical flow and vibration imaging. Compressive sensing serves as an efficient sampling method that requires significantly fewer frames than traditional approaches, selecting frames at less than one third of the original video. Features are extracted from the displacement of the compressed video frames. Subsequently, the vibration features are transformed into images. The proposed methodology is validated through experiments conducted on an air compressor. Comparative results between vibration imaging from video with compressive sensing and that from the original video demonstrate that vibrational features can be extracted using only a few frames of the original video. Consequently, this approach can reduce processing time and data volume while preserving the performance of vibration feature extraction.

Fast and accurate visual vibration measurement via derivative-enhanced phase-based optical flow

Visual vibration measurement using a digital camera is an emerging non-contact modality capable of enabling flexibility, high spatial resolution, and full field. However, the massive amounts of video data with vibration information (thousands of time-lapse images per measurement) to be handled lead to an actual bottleneck. This prevents the system from extracting the vibration information with high speed and fidelity. Here, a straightforward phase-based optical flow approach is proposed by introducing a derivative operation to perform vibration measurements in a more accurate and efficient manner, in contrast to the state-of-the-art phase-based visual method. This approach generates analytical signals by considering each raw gray image as the real part and its first-order derivative as the imaginary part. As a result, the physical vibration can be extracted from the phase variations of the analytical signals from one to the next. Numerical simulations and experiments are conducted to validate the effectiveness and accuracy of this simple but powerful approach. The results indicate that the proposed approach provides a significantly higher speed than current methods and a correlation coefficient of 99.2 % with the identification results.

Phase-Based Optical Flow Method with Optimized Parameter Settings for Enhancing Displacement Measurement Adaptability

Phase-Based Optical Flow Method with Optimized Parameter Settings for Enhancing Displacement Measurement Adaptability

Phase-based vibration imaging for structural dynamics applications: Marker-free full-field displacement measurements with confidence measures

Inspired by the concept of thermal imaging, we develop a phase-based vibration imaging technique using marker-free full-field displacement measurements, which aims to determine vibrational energy distribution and possibly vibration parameters as the thermal imaging determining thermal energy distribution using colormap. In this study, accurate full-field measurements are conducted at the sub-pixel level by performing a full-field phase-based optical flow with three novel progresses. First, an image-preprocessing Gaussian smoothing gradient is applied to reduce the nonlinear phase caused by homogenous patterns and image noise. Second, two confidence measures (phase nonlinearity and filter response) are adopted to obtain reliable displacement measurements estimated from the multi-scale and multi-direction phases. Third, the combination of vector average and the intersection of constraints ensure the accuracy of integrating every pixel’s displacement components. The vibrations can then be imaged by analyzing the acquired full-field displacements using techniques such as mean absolute deviation and frequency analyses. The capability of the proposed technique is demonstrated through numerical experiments using artificial patterns. After then, experimental validation is performed, which includes full-field operational deflection shape extraction on a 5-story structure and vibration imaging of an in-service hydrogen charging station, further proving the effectiveness of the proposed vibration imaging technique.

Informed pixel pushing: A new method of large-motion handling for phase-based optical flow

In this paper, a novel method is presented to increase the efficacy of phase-based optical flow (PBOF) in cases of simultaneous relevant super-pixel and sub-pixel motions. To date, PBOF performs well for sub-pixel motions of interest while measurement potential for super-pixel motion remains limited. Although current methods exist across the literature to handle irrelevant super-pixel motions otherwise obstructing sub-pixel extraction, they are typically aimed at removing the super-pixel motion. The proposed method does not assume irrelevancy, instead working to maintain the entire motion signal without resolution reduction or combined processing techniques. The objective of this work is to reframe the small motion assumption of optical flow through dynamic processing, steerable filters, and motion thresholds so that PBOF can accurately extract super-pixel motions – particularly large-scale translation and in-plane rotation – while still extracting sub-pixel motion. Proof of concept is presented through various synthetic test cases followed by experimental validation, all showing good agreement.

An approach for damage detection in noisy environments using DOG multi-scale space and fractal dimension

ABSTRACT The most prominent features of fractal dimension analysis are high sensitivity to damage and instant determination of damage location. However, an intrinsic deficiency is that it is difficult to identify damage in a structure while suppressing noise simultaneously. In addition, the vibration measurement with low-spatial-resolution is also the main factor restricting the accurate damage location. In this paper, a novel damage detection method based on fractal dimension is proposed, in order to provide the high-spatial-resolution vibration information, the phase-based optical flow is introduced, which is a vision-based measurement method that can calculate the structural vibration with high-spatial-resolution. Then, to detect damage in a structure while suppressing noise, an advanced signal analysis method is proposed, with this method, the fractal dimension (FRAC) is mapped into Difference of Gaussian (DOG) multi-scale space, producing the FRAC-DOG to reveal damage-induced singularities in mode shapes. Finally, a data fusion technique is applied to the proposed method to provide a reasonable damage detection result that maximises the elimination of uncertainty. The numerical and experimental results demonstrate that the proposed method can detect damage with high-precision in noisy environments by comparing it with the existing damage detection methods.

A robust structural vibration recognition system based on computer vision

Vibration-based structural health monitoring (SHM) systems are useful tools for assessing structural safety performance quantitatively. When employing traditional contact sensors, achieving high-resolution spatial measurements for large-scale structures is challenging, and fixed contact sensors may also lose dependability when the lifetime of the host structure is surpassed. Researchers have paid close attention to computer vision because it is noncontact, saves time and effort, is inexpensive, and has high efficiency in giving visual perception. In advanced noncontact measurements, digital cameras can capture the vibration information of structures remotely and swiftly. Thus, this work studies a system for recognizing structural vibration. The system ensures acquiring high-quality structural vibration signals by the following: 1) Establishing a novel image preprocessing, which includes visual partitioning measurement and image enhancement techniques; 2) initial recognition of structural vibration using phase-based optical flow estimation (POFE), which introduces 2-D Gabor wavelets to extract the independent phase information of the image to track the natural texture targets on the surface of the structure; 3) extracting the practical vibration information of the structure using mode decomposition to remove the complex environment of the camera vibration and other noises; 4) employing phase-based motion magnification (PMM) techniques to magnify small vibration signals, and then recognizing the complete information on the vibration time range of the structure. The research results of the laboratory experiments and field testing conducted under three different cases reveal that the system can recognize structural vibration in complicated environments.

Damage detection of structures from motion videos using high-spatial-resolution mode shapes and data fusion

Despite extensive research into damage detection based on mode shapes from vision-based methods in the past decades. There are still some notable insufficiencies, one of which is that vision-based damage detection requires a speckle pattern or mounting the high contrast markers on the surface, and the other is that damage detection accuracy in noisy environments is low, especially when detecting slight damage. In order to address these shortcomings, we proposed a high-precision damage detection strategy by combining an advanced vision-based measurement method with a signal analysis method. In terms of overcoming the requirement for a speckle pattern on the surface, a novel technique called phase-based optical flow is introduced to provide high-precision mode shapes. Then, considering the mode shape curvature is sensitive to damage feature, the Teager energy operator (TEO) together with wavelet transform (WT) to process the mode shape curvature, producing WT-TEO mode shape curvature to search for damage features in noisy environments. Finally, a data fusion algorithm based on Bayesian fusion theory is used to further eliminate the uncertain of noise interference by fusing the damage features across multi-scale space. The results of numerical and experiments demonstrate that the proposed strategy has the capability to detect single and multiple damages with high-precision in noisy environments.

Accuracy of phase-based optical flow for vibration extraction

In this paper, phase-based optical flow done using the complex steerable pyramid is evaluated to quantify upper and lower bounds of accurate pixel displacement extraction. The objective is to understand how typical image parameters affect these bounds and to determine if phase-based optical flow alone is effective and sufficient to extract vibrations. Across the current literature, phase-based optical flow is largely treated as a pre-processing step for more traditional optical methods, and its ability to extract motions rather than magnify them seems to have been largely overlooked. Rather than compound methodologies, phase-based optical flow is evaluated as a motion extraction tool in its own right in both a synthetic and experimental context. The evaluation provides expected bounds given ideal image parameters as well as a metric to adjust these bounds for a general scenario. Beyond these results, previous evaluations specific to phase-based optical flow are addressed and resulting bounds are compared. Due to dramatic effects caused by the image parameters, rules of thumb specific to this type of processing are also discussed.

마커 및 필터 설계를 통한 다양한 조도 조건에서의 영상 위상기반 2차원 변위 측정 기법

The advanced displacement measurement technique using computer vision has shown several advantages, such as a high spatial resolution and no mass loading effect, compared to the conventional contact-sensing techniques. However, the accuracy and robustness of vision-based techniques are subjected to various illumination conditions, including uneven illumination and insufficient lighting. This study introduced an accurate displacement measurement technique, with high robustness with respect to the illumination changes, that used two complex Gabor filters and a specially designed marker with orthogonal features. The parameters of the two complex Gabor filters, including the center frequency, bandwidth, and direction, were selected based on the marker size and attitude to extract the accurate phase motion information on two directions with dense features. The noise caused by insufficient light and/or uneven illumination could also be reduced by emphasizing the orthogonal features using the proper filter parameters. Further, motion phase-based optical flow was applied to calculate the 2D displacement signal of the marker. The excellent performance of displacement measurement was confirmed by comparing to that by a laser doppler vibrometer(LDV) under various lighting conditions. Tracking of multiple markers on a plate was performed to demonstrate the capability of measuring the out-plane motion using the proposed technique.

Phase-based displacement measurement on a straight edge using an optimal complex Gabor filter

In recent years, a number of phase-based motion processing techniques have been developed, and these allow for motion signals to be extracted from the local phase. The phase is generated by applying a pair of even- and odd-symmetric filters, and it is more robust to the intensity change. However, the phase can be unstable in certain areas where it is not linearly correlated with motion. This paper proposes an accurate and robust displacement measurement technique using an optimal complex Gabor filter and phase-based optical flow (POF) to measure the vibration response of a straight edge with the generated linear phase. The causes of nonlinear phase are discussed, and the nonlinearity of phase is used as an indicator of the measurement accuracy. Three types of nonlinear phase caused by noise background and inappropriate filter parameters for a straight edge are presented, along with solutions that optimize the parameters of the complex Gabor filter to obtain the phase with low nonlinearity. A numerical experiment is conducted on artificially created patterns to demonstrate the correlation between the proposed phase nonlinearity and the measurement accuracy. The validation experiments are carried out on a 5-story structure using a Laser Doppler Vibrometer (LDV) and a camera to demonstrate the accuracy and robustness of the proposed technique using the phase with low nonlinearity. The results show that the proposed technique can measure the vibration at 0.01-pixel scale. In addition, the robustness of the proposed technique is validated by performing experiments under different illumination conditions. The vibration responses of the 5-story structure with noisy background are measured using both accelerometers and a camera to perform the output-only modal identification. The results confirm that the proposed method can accurately identify modal frequencies, damping ratios, and high-resolution operational deflection shapes (ODS) requiring no pure background.

Identification of full-field dynamic modes using continuous displacement response estimated from vibrating edge video

The video of vibrating structure provides dense quantitative continuous spatial information that is harnessed using various computer vision algorithms in the past decade. Computer vision algorithms that implement sparse optical flow, have been successful in acquiring the Lagrangian representation of motion. Such vision-based measurement already proved its potential in replacing the need for a contact based vibration measurement sensors. In order to obtain full-field, spatially dense, vibrational modes, a large number of discrete sensors would be necessary throughout the specimen’s length, making it impractical. The phase-based video processing method is found to be successful in visualizing full-field operational mode shapes of a vibrating structure. However, the phase-based optical flow provides an Eulerian representation of the motion at every pixel of the image space, which does not acquire the full-field spatiotemporal Lagrangian displacement trajectory of the structure. Hence, there is a need to design a target-free, noncontact vision-based framework that can directly extract and quantify full-field dynamic modes of a real-life vibrating structural member from its video, by acquiring the trajectory of every particle on the structure at each frame of the video using optical flow. The continuous edge of a moving object is a rich optical feature whose motion perpendicular to its orientation can be tracked in Lagrangian coordinates using optical flow. In a recent paper by authors, the method of measuring full-field displacement response of a vibrating continuous edge of a structural member has been reported. In this paper, the full-field displacement response is acquired using the recently presented method and subsequently, its spatially dense dynamic modes are extracted from its video using the acquired full-field spatiotemporal displacement response of the vibrating structure. The full-field dynamic modes and modal parameters are extracted using the Hankel dynamic mode decomposition method, which is applicable both for linear, as well as nonlinear dynamical systems. Further, experimental validation of the proposed method is presented for two kinds of structures (1) a three-story steel frame, and (2) an aluminum cantilever beam, both undergoing free vibration. The results obtained using the proposed method is validated with the displacement measured using Laser Doppler Vibrometer at a particular point of the edge. The extracted modal properties using the proposed methodology compares satisfactorily with the results of the eigensystem realization algorithm applied to the discrete point accelerometer measurements attached at all three floors of the frame. Also, the numerically obtained mode shapes from the analytical model of the cantilever beam validate the estimated modal parameters and the full-field mode shapes. To validate the efficacy of the proposed method in real-world structures, part of the vibrating cable of Tacoma Narrows bridge is tracked from its video, moments before its collapse. The proposed method does successfully extract the dominating vibrational modes of the cables of the Tacoma Narrows bridge.

Output-only computer vision based damage detection using phase-based optical flow and unscented Kalman filters

A damage detection methodology is proposed by integrating a nonlinear recursive filter and a non-contact computer vision based algorithm to measure structural dynamic responses. A phase-based optical flow algorithm inspired by the motion magnification technique is used to measure structural displacements, and the unscented Kalman filter is used to predict structural properties such as stiffness and damping coefficients. This non-contact displacement measurement methodology does not require an intensive instrumentation process, does not add any additional mass to the structure which may skew measurements, and can measure more signals compared to traditional methods. This measurement methodology still needs improvement as a tool due to its higher noise level relative to traditional accelerometer and laser vibrometer measurements. In order to detect structural damage using measured displacements from video, an unscented Kalman filter is used to remove noise from the displacement measurement and simultaneously detect damage by identifying the current stiffness and damping coefficient values, given a known mass, which are used to detect damage. To validate the proposed damage detection method state-space equations are derived without external excitation input and experimental tests are carried out. The experimental results show reasonable and accurate predictions of the stiffness and damping properties compared to dynamic analysis calculations.

Analytic signal phase-based myocardial motion estimation in tagged MRI sequences by a bilinear model and motion compensation.

Different mathematical tools, such as multidimensional analytic signals, allow for the calculation of 2D spatial phases of real-value images. The motion estimation method proposed in this paper is based on two spatial phases of the 2D analytic signal applied to cardiac sequences. By combining the information of these phases issued from analytic signals of two successive frames, we propose an analytical estimator for 2D local displacements. To improve the accuracy of the motion estimation, a local bilinear deformation model is used within an iterative estimation scheme. The main advantages of our method are: (1) The phase-based method allows the displacement to be estimated with subpixel accuracy and is robust to image intensity variation in time; (2) Preliminary filtering is not required due to the bilinear model. The proposed algorithm, integrating phase-based optical flow motion estimation and the combination of global motion compensation with local bilinear transform, allows spatio-temporal cardiac motion analysis, e.g. strain and dense trajectory estimation over the cardiac cycle. Results from 7 realistic simulated tagged magnetic resonance imaging (MRI) sequences show that our method is more accurate compared with state-of-the-art method for cardiac motion analysis and with another differential approach from the literature. The motion estimation errors (end point error) of the proposed method are reduced by about 33% compared with that of the two methods. In our work, the frame-to-frame displacements are further accumulated in time, to allow for the calculation of myocardial Lagrangian cardiac strains and point trajectories. Indeed, from the estimated trajectories in time on 11 in vivo data sets (9 patients and 2 healthy volunteers), the shape of myocardial point trajectories belonging to pathological regions are clearly reduced in magnitude compared with the ones from normal regions. Myocardial point trajectories, estimated from our phase-based analytic signal approach, seem therefore a good indicator of the local cardiac dynamics. Moreover, they are shown to be coherent with the estimated deformation of the myocardium.

Registration accuracy and image quality of time averaged mid-position CT scans for liver SBRT

Purpose and backgroundThe purpose was to validate the accuracy of motion models derived from deformable registration from four-dimensional computed tomography (4DCT) and breath-hold contrast enhanced computed tomography (BHCCT) scans for liver SBRT. Additionally, the image quality of the time averaged mid-position (MidP) CT constructed using the detected motion model was assessed. Materials and methods4DCT and BHCCT liver scans of 11 patients were acquired with 1 or 2 fiducial markers. Using parametric sampling the markers were digitally removed. Phase-based optical flow was used to register the 4D frames and the BHCCT, and create MidP data. We compared the deformable registration of the markerless scans with the actual displacement of the markers to assess registration accuracy. The noise levels of the MidP scans were compared to those of the 4DCT and BHCCT data. ResultsWe found an average misregistration of 1.8mm (±0.5mm). The constructed MidPCT scan contained around three times less noise than the original 4D scan. The residual error between the MidPCT and the BHCCT was 3.0mm (±0.9mm). ConclusionsHigh precision deformable image registration of 4DCT and BHCCT liver cancer patients was achieved and used to create motion compensated MidPCT scans, with increased contrast-to-noise (CNR) levels. This improved visualisation of tumours and anatomy, facilitates radiotherapy treatment planning.

A Comparison of FPGA and GPU for Real-Time Phase-Based Optical Flow, Stereo, and Local Image Features

Low-level computer vision algorithms have extreme computational requirements. In this work, we compare two real-time architectures developed using FPGA and GPU devices for the computation of phase-based optical flow, stereo, and local image features (energy, orientation, and phase). The presented approach requires a massive degree of parallelism to achieve real-time performance and allows us to compare FPGA and GPU design strategies and trade-offs in a much more complex scenario than previous contributions. Based on this analysis, we provide suggestions to real-time system designers for selecting the most suitable technology, and for optimizing system development on this platform, for a number of diverse applications.

An Adapted Optical Flow Algorithm for Robust Quantification of Cardiac Wall Motion From Standard Cine-MR Examinations

This paper presents a method for local myocardial motion estimation from a conventional steady-state free precession cine-MRI sequence using a modified phase-based optical flow (OF) technique. Initially, the technique was tested on synthetic images to evaluate its robustness with regards to Rician noise and to brightness variations. The method was then applied to cardiac images acquired on 11 healthy subjects. Myocardial velocity is measured in centimeter per second in each studied pixel and visualized as colored vectors superimposed on MRI images. The estimated phase-based OF results were compared with a reference OF method and gave similar results on synthetic images, i.e., without a significant difference of the mean angular error. Applied on cine-MRI of normal hearts, the calculated velocities from short-axis images concord with values obtained in the literature. The advantage of the presented method is its robustness with respect to Rician noise and to brightness changes often observed in cine-MRI sequences, and especially with the through-plane movement of the heart. Motion assessment using our method on cine-MR images gives promising results on motion estimation on a pixel-by-pixel basis, leading to a regional measurement of the time-velocity course of myocardial displacement in different segments of the heart wall.

The Study of Phase-Based Optical Flow Technique Using an Adaptive Bilateral Filter

The purpose of this study is to propose an advanced phase-based optical flow method with improved tracking accuracy for motion flow. The proposed method is mainly based on adaptive bilateral filtering (ABF) and Gabor based spatial filtering. ABF aims to preserve the maximum boundary information of the original image, while the spatial filtering aims to accurately compute the local variations. Our method tracks the optical flow in three stages. Firstly, the input images are filtered by using ABF and a spatial filter to remove noises and to preserve the maximum contour information. The component velocities are then computed based on the phase gradient of each pixel. Secondly, irregular pixels are eliminated, if the phase differences are not linear over the image frames. Lastly, the entire velocity is derived by integrating the component velocities of each pixel. In order to evaluate the tracking accuracy of the proposed method, we have examined its performance for synthetic and realistic images for which the ground truth data were known. As a result, it was observed that the proposed technique offers higher accuracy than the existing optical flow methods.

Velocity field measurement of a round jet using quantitative schlieren

This paper utilizes the background oriented schlieren (BOS) technique to measure the velocity field of a variable density round jet. The density field of the jet is computed based on the light deflection created during the passage of light through the understudy jet. The deflection vector estimation was carried out using phase-based optical flow algorithms. The density field is further exploited to extract the axial and radial velocity vectors with the aid of continuity and energy equations. The experiment is conducted at six different jet-exit temperature values. Additional turbulence parameters, such as velocity variance and power spectral density of the vector field, are also computed. Finally, the measured velocity parameters are compared with the hot wire anemometer measurements and their correlation is displayed.