Optical Flow Estimation Method Research Articles

Development of a body movement detection system to avoid re-exposure during radiography.

During the radiographic examination of the chest and bones in hospitals, communicating and maintaining posture is difficult for some patients, and movement before or during X-ray irradiation may necessitate re-exposure owing to body wobbling movements or breathing movements. To prevent the need for re-exposure during radiography and to determine the exposure timing, a body movement detection system that considers breathing movements was developed in this study. The posture of a patient was monitored using an RGB camera. The acquired video data was analyzed to detect body movement using either an inter-frame difference method or an optical flow estimation method. The performance of the system was evaluated by detecting the body and breathing movements during positioning. Consequently, the inter-frame difference method detected 179.8-1222.2pixels during body movements, and the optical flow estimation method confirmed that the feature points moved by 5.5-26.6mm (4.2-20.3pixels). When detecting breathing movements, 82-585pixels were detected by the inter-frame difference method, and the optical flow estimation method showed that the feature points moved by 5.2mm (2-4pixels). Therefore, the proposed method can detect body movements during radiography to prevent re-exposure due to body wobble and breathing movements. For healthcare providers, it will lead to reduce not only concerns about patient exposure but also unnecessary radiographic workload.

PDTE: Pyramidal deep Taylor expansion for optical flow estimation

Optical flow estimation is an important hot research in computer vision. Although existing methods had got a considerable progress in improving their performance, they still have drawbacks, such as heavily computational burden, inaccurate pixel-level offset estimation, and poor interpretability. To address these issues, this letter proposes a pyramidal deep Taylor expansion (PDTE) framework, including: First, we seriously interpreted the relationship between optical flow computation and Taylor expansion. Then, the proposed PDTE is constructed by employing global motion aggregation (GMA) to calculate each derivative part which contributes to the final estimated optical flow. Quantitative and qualitative results on the Sintel and KITTI datasets validate that the proposed PDTE scheme is effective and outperforms the state-of-the-art optical flow estimation methods. The results of extensive experiments in the ablation study demonstrate that PDTE performs well on shape preservation and the accuracy improvement of optical flow estimation, even pixel-level offset calculation.

Determining dense velocity fields for fluid images based on affine motion.

In this article, we address the problem of estimating fluid flows between two adjacent images containing fluid and non-fluid objects. Typically, traditional optical flow estimation methods lack accuracy, because of the highly deformable nature of fluid, the lack of definitive features, and the motion differences between fluid and non-fluid objects. Our approach captures fluid motions using an affine motion model for each small patch of an image. To obtain robust patch matches, we propose a best-buddies similarity-based method to address the lack of definitive features but many similar features in fluid phenomena. A dense set of affine motion models was then obtained by performing nearest-neighbor interpolation. Finally, dense fluid flow was recovered by applying the affine transformation to each patch and was improved by minimizing a variational energy function. Our method was validated using different types of fluid images. Experimental results show that the proposed method achieves the best performance.

A Deep Learning Model for Precipitation Nowcasting Using Multiple Optical Flow Algorithms

Abstract The optical flow technique has advantages in motion tracking and has long been employed in precipitation nowcasting to track the motion of precipitation fields using ground radar datasets. However, the performance and forecast time scale of models based on optical flow are limited. Here, we present the results of the application of the deep learning method to optical flow estimation to extend its forecast time scale and enhance the performance of nowcasting. It is shown that a deep learning model can better capture both multispatial and multitemporal motions of precipitation events compared with traditional optical flow estimation methods. The model comprises two components: 1) a regression process based on multiple optical flow algorithms, which more accurately captures multispatial features compared with a single optical flow algorithm; and 2) a U-Net-based network that trains multitemporal features of precipitation movement. We evaluated the model performance with cases of precipitation in South Korea. In particular, the regression process minimizes errors by combining multiple optical flow algorithms with a gradient descent method and outperforms other models using only a single optical flow algorithm up to a 3-h lead time. Additionally, the U-Net plays a crucial role in capturing nonlinear motion that cannot be captured by a simple advection model through traditional optical flow estimation. Consequently, we suggest that the proposed optical flow estimation method with deep learning could play a significant role in improving the performance of current operational nowcasting models, which are based on traditional optical flow methods. Significance Statement The purpose of this study is to improve the accuracy of short-term rainfall prediction based on optical flow methods that have been employed for operational precipitation nowcasting. By utilizing open-source libraries, such as OpenCV, and commonly applied machine learning techniques, such as multiple linear regression and U-Net networks, we propose an accessible model for enhancing prediction accuracy. We expect that the improvement in prediction accuracy will significantly improve the practical application of operational precipitation nowcasting.

An optical flow estimation method based on multiscale anisotropic convolution

An optical flow estimation method based on multiscale anisotropic convolution

Real-Time Optical Flow Estimation Method Based on Cross-Stage Network

In this paper, a real-time optical flow estimation method based on a cross-stage network is proposed. The proposed model is designed with a network structure with encoders and decoders. The proposed method combines cross-stage network technology with the network structure of FlowNet2 and RAFT to achieve improved parameter number and estimation performance. For real-time optical flow estimation, it is important to maintain performance while reducing the number of parameters in the network. In the proposed method, structural convergence is performed to increase performance while reducing the number of parameters by applying the cross-stage network structure. The proposed model is designed to solve the bottlenecks in model accuracy and complexity by separating feature extraction and flow estimation processes. Flying Chairs, Flying Things 3D, and KITTI datasets were used to evaluate the performance of the proposed model, and the experimental results show superior performance compared to previous traditional methods.

A Review of Using Visual Odometery Methods in Autonomous UAV Navigation in GPS-Denied Environment

Abstract This review paper centers on strategies employed for location determination in regions lacking GPS signals. It primarily explores a range of vision-based methods tailored for this purpose, categorizing them accordingly. The article delves into the utilization of optical flow for feature extraction-based Visual Odometry (VO) and delves into advanced optical flow estimation methods that hinge on deep learning techniques. It compares the efficacy and practical applications of frequently utilized visual localization methods while also checking the efficiency of previous researches by reapplying the algorithms to new data and comparing the results.

SSTM: Spatiotemporal recurrent transformers for multi-frame optical flow estimation

Inaccurate optical flow estimates in and near occluded regions, and out-of-boundary regions are two of the current significant limitations of optical flow estimation algorithms. Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence. While this approach gives good flow estimates, it fails to generalize optical flows in occluded regions mainly due to limited local evidence regarding moving elements in a scene. In this work, we propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences. Our underlying hypothesis is that by understanding temporal scene dynamics from longer sequences with more than two frames, we can characterize pixel-wise dependencies in a larger spatiotemporal domain, generalize complex motion patterns and thereby improve the accuracy of optical flow estimates in occluded regions. We present learning-based spatiotemporal recurrent transformers for multi-frame based optical flow estimation (SSTMs). Our method utilizes 3D Convolutional Gated Recurrent Units (3D-ConvGRUs) and spatiotemporal transformers to learn recurrent space–time motion dynamics and global dependencies in the scene and provide a generalized optical flow estimation. When compared with recent state-of-the-art two-frame and multi-frame methods on real world and synthetic datasets, performance of the SSTMs were significantly higher in occluded and out-of-boundary regions. Among all published state-of-the-art multi-frame methods, SSTM achieved state-of the-art results on the Sintel Final and KITTI2015 benchmark datasets. Software code, data and instructions: https://github.com/Computational-Ocularscience/SSTM.

Physics-based optical flow estimation under varying illumination conditions

The physics-based optical flow (PBOF) model was derived from typical flow visualizations and it provided a more solid physical foundation for optical flow calculation. However, like many recent variational optical flow methods, it is not formulated to deal with the effect of illumination variation. In this paper, we propose a new efficient illumination-invariant optical flow estimation method, which improves the physics-based optical flow model. The proposed method introduces a multiplicative product of the velocity gradient and intensity from PBOF and an additive diffusion component to relax the brightness constancy assumption. The proposed method employs a linear transformation to reduce the influence of varying illumination on the data term. Then, a smoothness-sparsity regularization is used to constrain the minimization problem to have good edge-preserving for optical flow. Furthermore, the numerical implementation is given to solve the optimization problem. The accuracy of the developed method and the robustness against varying illumination are evaluated. The proposed method can provide motion estimation with acceptable accuracy for three optical flow datasets and cloud images of Jupiter’s great red spot, which have challenging illumination variations.

Applicability of deep learning optical flow estimation for PIV methods

Recently, there have been some attempts to introduce the method of deep learning-based optical flow(DL-based OF) into PIV measurements. These attempts have made it possible to estimate the flow field with higher accuracy and higher spatial resolution. The current mainstream deep-learning optical flow estimation method often requires a training set whose data distribution is closer to the real sample when completing the target task. However, there is a certain difference between the rigid body motion dataset commonly used in the field of computer vision and the fluid motion dataset in PIV measurements. Whether networks trained with rigid-body data can be directly applied to fluids has not been fully discussed. However, in existing attempts to introduce the DL-based OF method into the field of PIV measurement, the influence of the characteristics of the PIV particle image on the calculation results of the deep learning optical flow estimation (DL-based OF) has not been fully discussed. Therefore, the boundary conditions of the learning method for PIV calculation cannot be fully demonstrated. In this study, we selected the Flownet2 network, which is a classical baseline model on rigid body datasets, as a representative deep learning optical flow estimation method for experiments. Based on the characteristics of various particle images, this study generates a dataset with a wide range of parameters, such as particle density, particle diameter, sheet thickness, and maximum displacement, which are key to the generation of PIV simulation images and discusses the sensitivity of DL-based OF methods under extreme parameter conditions. This study compares (1) the model parameters trained using rigid body datasets such as Sintel, (2) the model parameters trained using synthetic PIV images, (3) the PIV-WIDIM method, and (4) the Optical Flow-Farneback method. The results reveal that the DL-based OF methods in (1) and (2) have a higher spatial resolution and estimation accuracy than the traditional methods in (3)] and (4). Training with the synthetic fluid dataset results in a smoother flow field in method (2)] compared with training with the rigid body dataset in method (1). In the case of limited training set parameters, Flownet2 can still be suitable for extreme conditions, such as small particle concentration, small particle diameter, insufficient particle brightness, or small particle displacement.

LCIF-Net: Local criss-cross attention based optical flow method using multi-scale image features and feature pyramid

Although CNN-based optical flow methods have achieved remarkable performance in terms of computational accuracy and efficiency, the issue of edge-blurring caused by large displacements remains an open challenge. To address this problem, we propose a local criss-cross attention based optical flow estimation method using multi-scale image features and feature pyramid. First, we design an image pyramid-based feature extraction sub-network and then incorporate it into the feature pyramid network to construct a hybrid feature extraction module, which is able to extract multi-scale structural and semantic information from the input images. Second, we concatenate a local criss-cross attention module with the hybrid feature extraction module to build a global feature encoder. The global feature encoder further captures the long-range dependencies within the feature map to improve the large displacement estimation performance. Finally, we combine the global feature encoder with an iterative optical flow decoder, and thus propose a novel network named LCIF-Net. We demonstrate its significant performance benefits on MPI-Sintel and KITTI datasets. Compared with the existing optical flow estimation methods, our LCIF-Net remarkably improves the accuracy and robustness for the optical flow estimation, especially in the regions with large displacements and motion edges.

Test-Time Adaptation for Optical Flow Estimation Using Motion Vectors.

Due to the prohibitive cost as well as technical challenges in annotating ground-truth optical flow for large-scale realistic video datasets, the existing deep learning models for optical flow estimation mostly rely on synthetic data for training, which in turn may lead to significant performance degradation under test-data distribution shift in real-world environments. In this work, we propose the methodology to tackle this important problem. We design a self-supervised learning task for adjusting the optical flow estimation model at test time. We exploit the fact that most videos are stored in compressed formats, from which compact information on motion, in the form of motion vectors and residuals, can be made readily available. We formulate the self-supervised task as motion vector prediction, and link this task to optical flow estimation. To the best of our knowledge, our Test-Time Adaption guided with Motion Vectors (TTA-MV), is the first work to perform such adaptation for optical flow. The experimental results demonstrate that TTA-MV can improve the generalization capability of various well-known deep learning methods for optical flow estimation, such as FlowNet, PWCNet, and RAFT.

Exploiting Discriminative Advantage of Spectrum for Hyperspectral Image Classification: SpectralFormer Enhanced by Spectrum Motion Feature

As for hyperspectral images (HSIs), the discrepancy of contiguous spectral information should be the main basis for the identification of ground objects. Due to the difficulty of spectral sequence coding and the spectrum similarity between categories, successful deep-learning-based classification methods always attempt to capture the spatial information to improve the accuracy by convolutional neural networks (CNNs) or other excellent spatial feature extractors. However, extracting spatial features is generally accompanied by the distortion of ground objects distribution and categories boundary. To effectively represent spectral features, the SpectralFormer based on transformer backbone can better capture the long-term dependence of the spectrum, which improves the performance of spectral feature methods significantly. However, it is still unable to compete with advanced spectral–spatial feature methods. To exploit the discriminative advantage of the spectrum fully, this letter introduces an efficient sparse-to-dense optical flow estimation method to track the spectrum variation in the HSI. Then, we take such a variation as a spectrum motion feature to enhance the original spectrum. At last, we continue to use the SpectralFormer to encode the concatenated spectrum sequence for classification. Extensive experiments show that the SpectralFormer enhanced by the spectrum motion feature (SF-SMF) significantly improves the performance of spectral feature methods, even surpassing advanced spectral–spatial feature methods. SF-SMF can avoid interference with additional spatial information to obtain exquisite whole-domain classification maps, showing its practical value. The codes will be public at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sssssyf/SF-SMF</uri> .

Robust optical flow estimation method based on structure-texture aware retinex model and its application on face anti-spoofing

Robust optical flow estimation method based on structure-texture aware retinex model and its application on face anti-spoofing

From Video to Hyperspectral: Hyperspectral Image-Level Feature Extraction with Transfer Learning

Hyperspectral image classification methods based on deep learning have led to remarkable achievements in recent years. However, these methods with outstanding performance are also accompanied by problems such as excessive dependence on the number of samples, poor model generalization, and time-consuming training. Additionally, the previous patch-level feature extraction methods have some limitations, for instance, non-local information is difficult to model, etc. To solve these problems, this paper proposes an image-level feature extraction method with transfer learning. Firstly, we look at a hyperspectral image with hundreds of contiguous spectral bands from a sequential image perspective. We attempt to extract the global spectral variation information between adjacent spectral bands by using the optical flow estimation method. Secondly, we propose an innovative data adaptation strategy to bridge the gap between hyperspectral and video data, and transfer the optical flow estimation network pre-trained with video data to the hyperspectral feature extraction task for the first time. Thirdly, we utilize the traditional classifier to achieve classification. Simultaneously, a vote strategy combined with features at different scales is proposed to improve the classification accuracy further. Extensive, well-designed experiments on four scenes of public hyperspectral images demonstrate that the proposed method (Spe-TL) can obtain results that are competitive with advanced deep learning methods under various sample conditions, with better time effectiveness to adapt to new target tasks. Moreover, it can produce more detailed classification maps that subtly reflect the authentic distribution of ground objects in the original image.

River Surface Velocity Estimation Using Optical Flow Velocimetry Improved With Attention Mechanism and Position Encoding

River surface velocity (RSV) estimation is crucial in adjusting water allocation, and preventing flood. The intrusive measurement equipment of RSV, which has insufficient real-time performance and a narrow detection range, cannot meet the needs for large-scale system deployment. In contrast, image-based velocimetry methods require no direct contact with river and can acquire almost real-time velocity information within the whole field of view of shooting camera. However, image-based velocimetry methods, such as current optical flow velocimetry (OFV), suffer from poor measurement accuracy in river scenes where massive optical noise from reflection and flickering light exists. To overcome this defect, a neural-network-based optical flow estimation model which adopts a framework composed of feature extraction and optical flow optimization is proposed. Attention mechanism and position encoding are employed in the optical flow optimization process to improve the estimation performance in complex optical conditions. As results, the proposed method achieves an average RSV estimation accuracy of 92.03% in normal scenes and maintains a low average relative error of only 11.61% even in scenes with complex optical noise. We believe that this work delineates the perspective of RSV in deep optical flow estimation method and prompt the application of water-related disaster warning.

ASFlow: Unsupervised Optical Flow Learning With Adaptive Pyramid Sampling

We present an unsupervised optical flow estimation method by proposing an adaptive pyramid sampling in the deep pyramid network. Specifically, in the pyramid downsampling, we propose a Content-Aware Pooling (CAP) module, which promotes local feature gathering by avoiding cross region pooling, so that the learned features become more representative. In the pyramid upsampling, we propose an Adaptive Flow Upsampling (AFU) module, where cross edge interpolation can be avoided, producing sharp motion boundaries. Equipped with these two modules, our method achieves the best performance for unsupervised optical flow estimation on multiple leading benchmarks, including MPI-Sintel, KITTI 2012 and KITTI 2015. Particularly, we achieve EPE=1.5 on KITTI 2012 and F1=9.67% KITTI 2015, which outperform the previous state-of-the-art methods by 16.7% and 13.1%, respectively.

Unsupervised optical flow estimation method based on transformer and occlusion compensation

Unsupervised optical flow estimation method based on transformer and occlusion compensation

Real-time dense-view imaging for three-dimensional light-field display based on image color calibration and self-supervised view synthesis.

Three-Dimensional (3D) light-field display has achieved promising improvement in recent years. However, since the dense-view images cannot be collected fast in real-world 3D scenes, the real-time 3D light-field display is still challenging to achieve in real scenes, especially at the high-resolution 3D display. Here, a real-time 3D light-field display method with dense-view is proposed based on image color correction and self-supervised optical flow estimation, and a high-quality and high frame rate of 3D light-field display can be realized simultaneously. A sparse camera array is firstly used to capture sparse-view images in the proposed method. To eliminate the color deviation of the sparse views, the imaging process of the camera is analyzed, and a practical multi-layer perception (MLP) network is proposed to perform color calibration. Given sparse views with consistent color, the optical flow can be estimated by a lightweight convolutional neural network (CNN) at high speed, which uses the input image pairs to learn the optical flow in a self-supervised manner. With inverse warp operation, dense-view images can be synthesized in the end. Quantitative and qualitative experiments are performed to evaluate the feasibility of the proposed method. Experimental results show that over 60 dense-view images at a resolution of 1024 × 512 can be generated with 11 input views at a frame rate over 20 fps, which is 4× faster than previous optical flow estimation methods PWC-Net and LiteFlowNet3. Finally, large viewing angles and high-quality 3D light-field display at 3840 × 2160 resolution can be achieved in real-time.

Robust depth estimation on real-world light field images using Gaussian belief propagation

Depth estimation on light field images has been well researched on synthetic datasets. However, existing methods often fail on real-world light field images because of the real-world noises. Due to the lack of real-world light field image datasets for depth estimation, the learning-based methods, e.g., convolutional neural networks, are not capable of performing well on real-world light field images with only synthetic datasets available. In this paper, we adopt the optical flow estimation method to estimate depth from light field images since (i) the existing optical flow methods have shown robustness with significant real-world noises; (ii) the adopted optical flow estimation method does not require any training data, thus not limited to the synthetic datasets. The depth can be solved via the approach of optical flow estimation because (i) a sub-aperture image array can be converted from a light field image; (ii) the optical flow between every two adjacent sub-aperture images of the image array is exactly the disparity between the two images. Furthermore, since a sub-aperture array contains numerous adjacent image pairs, numerous optical flow maps can be generated from the sub-aperture array. We show the depth can be well refined from these numerous optical flow maps through modeling these maps in a single graphical model (a pairwise Gaussian MRF to be precise) and inferencing this graphical model by Gaussian belief propagation (GaBP). Experiments show that i) the proposed method achieves better depth estimation than the state-of-the-art methods on real-world light field images; ii) the proposed method can estimate the depth of objects at a far distance (3.5–9.5 m) much more accurately than the state-of-the-art convolutional neural networks.