Space-time Image Research Articles

Space-time image velocimetry in blurred scenes based on BSTI-DCGAN data augmentation

Due to the limited sample quantity and the complex data collection process of the blurred space-time image (BSTI) dataset, the deep learning-based space-time image velocimetry (STIV) results in larger errors when applied to blurry videos. To enhance the measurement accuracy, we propose the use of STIV in blurred scenes based on BSTI-deep convolutional generative adversarial network (DCGAN) data augmentation. Firstly, BSTI-DCGAN is developed based on the DCGAN. This network utilizes a bilinear interpolation-convolution module for upsampling and integrates coordinated attention and multi-concatenation attention to enhance the resemblance between generated and real images. Next, further expanding the dataset by using artificially synthesized space-time images subsequently, all space-time images are transformed into spectrograms to create a training dataset for the classification network. Finally, the primary spectral direction is detected using the classification network. The experimental results indicate that our approach effectively augments the dataset and improves the accuracy of practical measurements. Under the condition of video blur, the relative errors of the average flow velocity and discharge are 3.92% and 2.72%, respectively.

A two‐dimensional, reach‐scale implementation of space‐time image velocimetry (STIV) and comparison to particle image velocimetry (PIV)

AbstractImage‐based algorithms have become a powerful tool for estimating flow velocities in rivers. In this study, we generalize the space‐time image velocimetry (STIV) framework for reach‐scale application rather than along a cross section. The new algorithm provides information on both the magnitude and orientation of velocity vectors, and we refer to the algorithm as two‐dimensional STIV, or 2D‐STIV. The workflow involves setting up a grid, using centreline tangent vectors as initial estimates of flow direction, and then extracting space‐time images (STIs) along search lines radiating from each grid node. The autocorrelation function is used to infer the inclination of streak lines present in STIs, which represents the advection of water surface features. Information on flow direction is obtained by evaluating various candidate search lines and identifying that which yields the highest velocity. This search can be performed exhaustively or via optimization. We applied the new 2D‐STIV algorithm to three test cases, one simulated data set and two natural channels, and compared image‐derived velocities to modelled or measured values. We also applied two established particle image velocimetry (PIV) algorithms to the same data sets. 2D‐STIV performed as well as the two PIV algorithms for simulated images. For a natural river with distinct water surface features, 2D‐STIV was effective for much of the channel but also led to a more patchy, irregular velocity field than the two PIV algorithms. For a site lacking obvious surface features, exhaustive 2D‐STIV led to velocity estimates uncorrelated with field data while the optimization‐based version produced erratic flow directions. 2D‐STIV also required greater image sequence durations, higher frame rates, and generally longer computational run times. Overall, ensemble PIV was the most reliable algorithm.

Optimization of Space-Time image velocimetry based on deep residual learning

Space-Time Image Velocimetry (STIV) is considered to be a highly promising method for river flow measurement due to its safe and efficient features. However, the traditional STIV method is challenged to obtain stable and accurate results when disturbed by external environments such as obstacles, waves and noise. As a consequence, a residual network-based STIV method is constructed by combining STIV with deep learning. In the synthetic dataset validation, benefited from the unique residual module and the relation-aware global attention mechanism, the recognition accuracy of STI texture angle is as high as 99.9 %, which is much higher than that of the Efficient Net b3, ViT16 and VGG19 networks. In the real dataset, the proposed method shows stronger stability and lower error by compared with gradient tensor (GT) and fast Fourier transform (FFT) methods, whose overall recognition accuracy is above 90 % under all types of STI cases, angle error is within 2°, and flow velocity error is around 0.10 m/s. Furthermore, the model was validated by a one-year comparative measurement at the Qilijie hydrological station in 2022, and the results were compared with LSPIV, GT, and FFT methods. The R2 of the model to the measured values is 0.98, the RMSE is 0.17, and the MAE is 0.12 m/s, indicating that the method has sufficient accuracy and robustness for long-term stable and real-time monitoring of river flow.

A method of applying deep learning based optical flow algorithm to river flow discharge measurement

River flow discharge monitoring is one of the critical tasks performed at hydrological stations. The large-scale particle image velocimetry (LSPIV) method widely used in hydrological stations is often limited by a lack of floating objects and has a high computational complexity. The space-time image velocimetry method is susceptible to noise interference and requires high stability of the flow over time. This paper proposes a flow measurement method based on the recurrent all-pairs field transforms for optical flow (RAFT) algorithm. The 4D correlation volume pyramid in the RAFT network structure can effectively handle changing and complex flow conditions. The convolutional block attention module is introduced into the optical flow update module after the 4D correlation volume pyramid, enhancing the ability to capture complex flow surface information. Additionally, feature extraction adds deformable convolution to expand the receptive field of the flow image, which has better adaptability in non-rigid motion. To validate the effectiveness of the new method (RAFT-D-C), this paper conducts comparative experiments with both existing and new methods. The experimental results show that RAFT-D-C has relative errors of 2.13% and 4.41% for the average flow velocity of two rivers and relative errors of 2.19% and 3.05% for the total discharge, respectively. RAFT-D-C provides improved accuracy compared to other methods and requires less computational run time than the frequently used LSPIV method.

River video flow measurement algorithm with space-time image fusion of object tracking and statistical characteristics

River flow monitoring serves as a crucial data source in hydrological stations, employing the widely adopted space-time image velocimetry (STIV) method both domestically and internationally. However, current methods yield space-time (ST) image trajectories with limited usable information, particularly due to the interference of complex flow characteristics and harsh outdoor environments in natural rivers. This interference leads to errors in detecting the main orientation of texture, consequently reducing measurement accuracy. To overcome these challenges, this paper proposes an river video flow measurement algorithm with ST image fusion of object tracking and statistical characteristics. The Lucas–Kanade tracking algorithm is employed to track velocity points and generate tracking trajectories. These trajectories’ coordinates are then fitted into straight lines to automatically arrange velocity-measuring lines in natural rivers. The algorithm leverages multi-scale detail boosting and Gaussian directional stretch filtering to enhance the quality and texture trajectories of ST images. Furthermore, background noise filtering and stripe noise filtering are applied to denoise the ST images, effectively mitigating the impact of noise caused by the natural river environment and camera equipment. The proposed algorithm incorporates projection statistical characteristics algorithms to accurately detect the dominant texture direction, enabling precise calculation of mean velocity and cross-sectional discharge of the river. Experimental validation was conducted in both natural and regular river channels, demonstrating the superior performance of the proposed solution compared to widely used large-scale particle image velocimetry, STIV, MobileNet-STIV and FD-DIS-G algorithms. The algorithm exhibited higher accuracy, stability, and applicability, with errors at each velocity-measuring point below 10%, and relative errors of the calculated mean velocity and cross-sectional discharge below 3%. This affirms the algorithm’s superior accuracy, stability, and applicability.

Mutual validation of remote hydraulic estimates and flow model simulations using UAV-borne LiDAR and deep learning-based imaging techniques

Hydraulic modeling plays a pivotal role in various river-related applications, but acquiring precise datasets can be challenging due to complex environments and limited availability of field data. To address these challenges, remote sensing approaches have shown promise, but their widespread scientific use requires validation. This study investigates the utilization of an uncrewed aerial vehicle (UAV)-integrated topo-bathymetric green LiDAR system (GLS) combined with deep learning-based space-time image velocimetry (DL-STIV) techniques to remotely analyze hydraulic properties in the lower Asahi River, Japan. The research compares streamflow measurements with flow model simulations at four transects within a 1.2 km gravel bed channel with vegetation. The results demonstrate that the DL-STIV technique provides reliable velocity estimates, with root-mean-square errors ranging from 0.014 to 0.247 m/s. When using a typical velocity conversion factor of 0.85, the DL-STIV method aligns well with model predictions, yielding coefficient of determination values between 0.850 and 0.959. However, the choice of conversion index (ranging from 0.75 to 0.95) significantly affects accuracy. The study also reveals that remotely estimated discharges are reasonably consistent with model estimates, with errors ranging from 0.026 to 11.124%. However, accuracy is influenced by factors such as the presence of obstacles, uncertainties in remotely sensed water depths, and the choice of reference values. In conclusion, this research demonstrates the suitability and potential of GLS and DL-STIV for remotely analyzing hydraulic properties in rivers. The study also points to the importance of considering potential variations and uncertainties when adopting these techniques for flow modeling and validation purposes.

Adaptively monitoring streamflow using a stereo computer vision system

Abstract. The gauging of free surface flows in waterways provides the foundation for monitoring and managing the water resources of built and natural environments. A significant body of literature exists around the techniques and benefits of optical surface velocimetry methods to estimate flows in waterways without intrusive instruments or structures. However, to date, the operational application of these surface velocimetry methods has been limited by site configuration and inherent challenging optical variability across different natural and constructed waterway environments. This work demonstrates a significant advancement in the operationalisation of non-contact stream discharge gauging applied in the computer vision stream gauging (CVSG) system through the use of methods for remotely estimating water levels and adaptively learning discharge ratings over time. A cost-effective stereo camera-based stream gauging device (CVSG device) has been developed for streamlined site deployments and automated data collection. Evaluations between reference state-of-the-art discharge measurement technologies using DischargeLab (using surface structure image velocimetry), Hydro-STIV (using space–time image velocimetry), acoustic Doppler current profilers (ADCPs), and gauging station discharge ratings demonstrated that the optical surface velocimetry methods were capable of estimating discharge within a 5 %–15 % range between these best available measurement approaches. Furthermore, results indicated model machine learning approaches leveraging data to improve performance over a period of months at the study sites produced a marked 5 %–10 % improvement in discharge estimates, despite underlying noise in stereophotogrammetry water level or optical flow measurements. The operationalisation of optical surface velocimetry technology, such as CVSG, offers substantial advantages towards not only improving the overall density and availability of data used in stream gauging, but also providing a safe and non-contact approach for effectively measuring high-flow rates while providing an adaptive solution for gauging streams with non-stationary characteristics.

Velocity Vector Estimation of Two-Dimensional Flow Field Based on STIV

As an important part of hydrometry, river discharge monitoring plays an irreplaceable role in the planning and management of water resources and is an essential element and necessary means of river management. Due to its benefits of simplicity, efficiency and safety, Space-Time Image Velocimetry (STIV) has attracted attention from all around the world. The most crucial component of the STIV is the detection of the Main Orientation of Texture (MOT), and the precision of detection directly affects the results of calculations. However, due to the complicated river flow characteristics and the harsh testing environment in the field, a large amount of noise and interfering textures show up in the space-time images, which affects the detection results of the MOT. In response to the shortage of noise and interference texture, a new non-contact image analysis method is developed. Firstly, Multi-scale Retinex (MSR) is proposed to pre-process the images for contrast enhancement; secondly, a fourth-order Gaussian derivative steerable filter is employed to enhance the structure of the texture; next, based on the probability density distribution function and the orientations of the enhanced images, the noise suppression function and the orientation-filtering function are designed to filter out the noise to highlight the texture. Finally, the Fourier Maximum Angle Analysis (FMAA) is used to filter out the noise further and obtain the clear orientations to achieve the measurement of velocity and discharge. The experimental results show that, compared with the widely used image velocimetry measurements, the accuracy of our method in the average velocity and flow discharge is significantly improved, and the real-time performance is excellent.

Space-Time Image Velocimetry Based on Improved MobileNetV2

Space-time image velocimetry (STIV) technology has achieved good performance in river surface-flow velocity measurement, but the application in a field environment is affected by bad weather or lighting conditions, which causes large measurement errors. To improve the measurement accuracy and robustness of STIV, we combined STIV with deep learning. Additionally, considering the light weight of the neural network model, we adopted MobileNetV2 and improved its classification accuracy. We name this method MobileNet-STIV. We also constructed a sample-enhanced mixed dataset for the first time, with 180 classes of images and 100 images per class to train our model, which resulted in a good performance. Compared to the current meter measurement results, the absolute error of the mean velocity was 0.02, the absolute error of the flow discharge was 1.71, the relative error of the mean velocity was 1.27%, and the relative error of the flow discharge was 1.15% in the comparative experiment. In the generalization performance experiment, the absolute error of the mean velocity was 0.03, the absolute error of the flow discharge was 0.27, the relative error of the mean velocity was 6.38%, and the relative error of the flow discharge was 5.92%. The results of both experiments demonstrate that our method is more accurate than the conventional STIV and large-scale particle image velocimetry (LSPIV).

River Surface Velocimetry Based on Virtual River Dataset and Modulated-RAFT

Many image-based velocimetry methods, such as large-scale particle image velocimetry (LSPIV), space-time image velocimetry (STIV) and optical flow velocimetry (OFV), have been proposed to estimate river surface velocity (RSV) efficiently and precisely. Among these methods, OFV-based methods have drawn great attention due to its high field resolution and low requirement of tracers. As a powerful approach for estimate optical flow accurately and efficiently, deep optical flow estimation is utilized in OFV-based methods as well. However, these methods often use irrelevant datasets for training due to the immeasurability of optical flow. Models obtained by such approaches suffer from limited generalization because of domain drift. Besides, high similarity of river surfaces often leads to ambiguous correlation volume extracted by deep optical flow estimation model, which can cause mismatch. This work proposed a method for accurately and robustly RSV estimation under velocity range of 0-6.0 m/s. Specifically, we introduced a method for generating relative optical flow datasets and proposed MRAFT, a deep optical flow estimation model combining with correlation volume modulation. Experiment results demonstrate that the generated virtual river datasets effectively improve the generalization of the model and MRAFT remarkably alleviates the mismatch. Our work facilitates the application of deep optical flow on RSV estimation and provides other related researches with optical flow datasets for fine-tuning.

Three-dimensional numerical modelling of floods in river corridor with complex vegetation quantified using airborne LiDAR imagery

Excessive flood flow over the historic diversion weir in the vegetated Asahi River in Okayama Prefecture, Japan, was recently recorded for the first time after its renovation work. Fluvial researchers analysed the diversion discharge for flood mitigation measures through laboratory studies and conventional two-dimensional (2-D) depth-averaged simulations. The existing model was insufficient for simulation of certain phenomena such as flow resistance caused by vegetation branches and leaves and vertical flow distribution around the river corridor. Therefore, we developed a three-dimensional (3-D) vegetation resistance porous model by estimating topography, land cover, and vegetation distribution from airborne light detection and ranging (LiDAR) topo-bathymetry (ALB) data. Results show that the water level and flow regime were more reproducible than by referenced 2-D calculations when compared to space-time image velocimetry (STIV) data and field measurements. The diversion discharge designed using the proposed model is feasible with the current riverbed and vegetation conditions.

Promises and uncertainties in remotely sensed riverine hydro‐environmental attributes: Field testing of novel approaches to unmanned aerial vehicle‐borne lidar and imaging velocimetry

AbstractRecent advancements in remotely sensed techniques have markedly expanded data acquisition potential in riverine studies, but the techniques' applicability must be validated and improved because of uncertainties associated with diverse field conditions. This study is the first experimental evidence of using a newly designed unmanned aerial vehicle (UAV)‐borne green lidar system (GLS) and deep learning‐automated space–time image velocimetry (STIV) for remote investigation of hydraulic and vegetation quantities of the gravel‐bed Asahi River in Okayama Prefecture, Japan. In addition to identifying bed deformation in waters shallower than 2 m, the GLS point clouds characterized the submerged infrastructure with block detailing patterns, thereby identifying positional displacement and severely damaged parts. This paper also presents a noncontact method of estimating incremental river discharge. Compared to benchmarked flow model estimates, remotely sensed discharges for three transects covering shallower, deeper, and partially submerged woody vegetation areas were overestimated by 1–11%, with 4% underestimation for another cross‐section. The STIV analysis also showed complicated flow patterns that were reasonably confirmed by flow vectors from depth‐averaged modeling. Ultimately, depth‐averaged flow model estimates validated hydraulic parameters derived remotely from GLS and STIV, and vice versa. In addition to approximating vegetation growth rates, the study using GLS attributes accurately identified riparian vegetation types as herbaceous (70%), woody (86%), and bamboo groves (65%). Finally, our findings provide insight into the management of shallow clear‐flowing vegetated rivers and remote sensing of streamflow to validate hydrodynamic‐numerical methods.

Free-Surface Velocity Measurement Using Direct Sensor Orientation-Based STIV.

Particle image velocimetry (PIV) is a quantitative flow visualization technique, which greatly improves the ability to characterize various complex flows in laboratory and field environments. However, the deployment of reference objects or ground control points (GCPs) for velocity calibration is still a challenge for in situ free-surface velocity measurements. By combining space-time image velocimetry (STIV) with direct sensor orientation (DSO) photogrammetry, a laser distance meter (LDM)-supported photogrammetric device is designed, to realize the GCPs-free surface velocity measurement under an oblique shooting angle. The velocity calibration with DSO is based on the collinear equation, while the lens distortion, oblique shooting angle, water level variation, and water surface slope are introduced to build an imaging measurement model with explicit physical meaning for parameters. To accurately obtain the in situ position and orientations of the camera utilizing the LDM and its embedded tilt sensor, the camera’s intrinsic parameters and relative position within the LDM are previously calibrated with a planar chessboard. A flume experiment is designed to evaluate the uncertainty of optical flow estimation and velocity calibration. Results show that the proposed DSO-STIV has good transferability and operability for in situ measurements. It is superior to propeller current meters and surface velocity radars in characterizing shallow free-surface flows; this is attributed to its non-intrusive, whole-field, and high-resolution features. In addition, the combined uncertainty of free-surface velocity measurement is analyzed, which provides an alternative solution for error assessment when comparing measurement failures.

Image-based migration velocity and dune length in clear water rivers

This work presents a novel way of measuring image-based migration velocity and dune length in clear water rivers. These results, together with empirical formulations, can be used to estimate bedload sediment transport with a high-spatial resolution. Image processing relies on the non-intrusive image-based technique called Space-Time Image Velocimetry (STIV). STIV consists in visualizing pixel intensity variation in time over a defined line. This technique was originally applied to hydraulics in order to characterize streamwise surface velocities in rivers. In this case, STIV is used to measure dune migration velocity and length in a shallow river with clear water in Córdoba Province, Argentina. Image processing procedures are detailed, including pre-processing and rectification. Results are then compared with measurements obtained from Acoustic Doppler Current Profilers. Finally, as an appendix, using the measured data and empirical equations, an estimation of bedload sediment transport is performed using both techniques.

Measurement of debris flow velocity in flume using normal image by space-time image velocimetry incorporated with machine learning

Various methods have been applied to the velocity measurement in the soil and rocks waterway experiment. Although most velocity measurements use observation sensors, soil and rocks are two-phase flow in which water and soil are mixed, and the velocity measurement method using direct contact with soil and sensors has various problems, including the potential for equipment damage and noise. Therefore, a velocity measurement method that employs high-resolution image analysis using a high-speed camera, which is an indirect velocity measurement method, has been proposed. Although the measurement of the velocity using high-resolution image analysis is associated with a very little risk of equipment damage, several problems have been reported, such as expensive system construction costs, analysis methods to eliminate noise present in images, and the low accuracy of the velocity measurement according to the slope (velocity) evaluation by space–time image (STI), indicating the movement velocity for each frame (moving distance). This study aims to measure the velocity of soil and rocks by applying space–time image velocimetry (STIV), which has not yet been applied to the soil and rocks waterway experiment. To evaluate the STIV performance, an indoor waterway experiment was carried out, and the results were compared with the velocity measured using a general-purpose high-speed camera. Machine learning was applied to the STI slope estimation method to improve the performance of the velocity measurement according to the noise interference included in the image. The use of convolutional neural networks (CNNs) is a useful way of extracting the characteristics of images, and they have learned to detect slopes using STI's 2D Fourier transform images. As a result, the performance of the velocity evaluation was significantly improved. Furthermore, it is possible to easily measure the velocity with high accuracy by freely setting an area to be measured in the captured image, such as the approaching velocity of soil and rocks or the overflow velocity at a debris barrier.

Measurement of Debris Flow Velocity in Flume Using Normal Image by Space-Time Image Velocimetry Incorporated with Machine Learning

Measurement of Debris Flow Velocity in Flume Using Normal Image by Space-Time Image Velocimetry Incorporated with Machine Learning

Application of an Entropic Method Coupled with STIV for Discharge Measurement in Actual Rivers

In recent years, due to the frequent occurrence of floods that exceed the facility maintenance level due to climate change, non-contact flood flow measurement techniques have been paid attention and actually some measurements have been conducted by applying them instead of the conventional float method. The space-time image velocimetry (STIV) which can measure the surface flow velocity distribution from video images is one of such techniques. In order to calculate the river flow from the surface velocity distribution, it is necessary to determine an appropriate surface velocity coefficient, which is the ratio of the average depth velocity to the surface velocity. However, at present, empirical default value has been still used in practice. In this study, the cross-sectional velocity distribution was calculated using an entropic method by utilizing the surface velocity distribution measured by STIV and compared with Acoustic Doppler Current Profiler (ADCP) observation. It was confirmed that the introduction of the velocity dip system express the flow velocity distribution in the vertical direction, where the velocity dip occurs due to the influence of vegetation.

A photogrammetry-based variational optimization method for river surface velocity measurement

The ease of access to media resources and computational power has recently generated interest in using vision-based approaches for hydraulic monitoring. A key challenge for non-intrusive, image-based hydrology measurement methods is incorporating different hydraulic variables as prior knowledge with image information. We propose a photogrammetry-based method called L1-Diffusion derived from the convection–diffusion equation commonly used in hydrodynamics with an additional regularization term to estimate the fluid motion field in the image plane, from which the free surface velocity can be further obtained using the photogrammetric projection relationship between the image plane and world coordinates. The inverse problem is used to discuss the relationship between the widely used space–time image velocimetry (STIV) and the proposed L1-Diffusion. To validate the proposed method, unmanned aerial vehicle (UAV) images as well as in-situ acoustic Doppler current profiler (ADCP) experiments were carried out. Based on comparison results with the ADCP measurement and vision-based flow field estimation, the newly proposed L1-Diffusion algorithm can accurately and efficiently estimate the free surface velocity of a river from the image sequences in a variety of scenarios.

Evaluation of the Applicability of STIV to Wave Characteristic Measurement in the Swash Zone

The swash zone is an area that causes a change in the shape of a beach by generating sediment transport under the influence of intermittent waves, where wave run-up and run-down are infinitely repeated in the final stage of the shoaling process. However, the ability to predict the sediment transport is extremely poor despite the swash zone being an extremely important area in terms of offshore disaster prevention. In particular, many researchers are conducting studies on the development of various types of observation equipment and analysis techniques because the turbulent flow of active fluid dominates the sediment transport and is an extremely important parameter for the analysis of the transport mechanism. However, in flow velocity measurement, it is difficult to measure a quantitative representative flow velocity over time because the swash zone has a shallow water depth and an active turbulent flow. Expensive equipment and short-time measurement are also limitations. Therefore, the purpose of this study was to evaluate the applicability of nonintrusive space-time image velocimetry(STIV) to analyze the flow characteristics of fluid in the swash zone, such as the movement velocity and period of intermittent waves in the shoaling process. The prediction accuracy was improved by removing various noises included in the images with the introduction of artificial intelligence for immediate and accurate calculation of the representative flow velocity using images that can be obtained easily. Consequently, it was discovered that the spatial representative flow velocity occurring in the swash zone, change in the wave period according to the shoaling effect, rip current and surface velocity can be measured.

Improving Accuracy and Robustness of Space-Time Image Velocimetry (STIV) with Deep Learning

Image-based river flow measurement methods have been attracting attention because of their ease of use and safety. Among the image-based methods, the space-time image velocimetry (STIV) technique is regarded as a powerful tool for measuring the streamwise flow because of its high measurement accuracy and robustness. However, depending on the image shooting environment such as stormy weather or nighttime, the conventional automatic analysis methods may generate incorrect values, which has been a problem in building a real-time measurement system. In this study, we tried to solve this problem by incorporating the deep learning method, which has been successful in the field of image analysis in recent years, into the STIV method. The case studies for the three datasets indicated that deep learning can improve the efficiency of the STIV method and can continuously improve performance by learning additional data. The proposed method is suitable for building a real-time measurement system because it has no tuning parameters that need to be adjusted according to the shooting conditions and the calculation speed is fast enough for real-time measurement.