Optical Flow Field Research Articles

Seam estimation based on dense matching for parallax-tolerant image stitching

Image stitching with large parallax poses a significant challenge in the field of computer vision. Existing seam-based approaches attempt to address parallax artifacts by stitching images along seams. However, issues such as object mismatches, disappearances, and duplications still arise occasionally, primarily due to inaccurate alignment of dense pixels or inappropriate seam estimation methods. In this paper, we propose a robust seam-based parallax-tolerant image stitching method that leverages dense flow estimation from state-of-the-art approaches. Firstly, we develop a seam estimation method that does not require pre-estimation of image warping model. Instead, it directly estimates the seam by measuring the local smoothness of the optical flow field and incorporating a penalty term for duplications. Subsequently, we design an iterative algorithm that utilizes the location of estimated seam to solve a spatial smooth warping model and eliminate outlier corresponding pairs. By employing this approach, we effectively address the intertwined challenges of estimating the warping model and seam. Experiment on real-world images shows that our proposed method achieves superior local alignment accuracy near the stitching seam and outperforms other state-of-the-art techniques on visual stitching result. Code is available at https://github.com/zhihao0512/dense-matching-image-stitching.

Crowd Density Estimation via Global Crowd Collectiveness Metric

Drone-captured crowd videos have become increasingly prevalent in various applications in recent years, including crowd density estimation via measuring crowd collectiveness. Traditional methods often measure local differences in motion directions among individuals and scarcely handle the challenge brought by the changing illumination of scenarios. They are limited in their generalization. The crowd density estimation needs both macroscopic and microscopic descriptions of collective motion. In this study, we introduce a Global Measuring Crowd Collectiveness (GMCC) metric that incorporates intra-crowd and inter-crowd collectiveness to assess the collective crowd motion. An energy spread process is introduced to explore the related crucial factors. This process measures the intra-crowd collectiveness of individuals within a crowded cluster by incorporating the collectiveness of motion direction and the velocity magnitude derived from the optical flow field. The global metric is adopted to keep the illumination-invariance of optical flow for intra-crowd motion. Then, we measure the motion consistency among various clusters to generate inter-crowd collectiveness, which constitutes the GMCC metric together with intra-collectiveness. Finally, the proposed energy spread process of GMCC is used to merge the inter-crowd collectiveness to estimate the global distribution of dense crowds. Experimental results validate that GMCC significantly improves the performance and efficiency of measuring crowd collectiveness and crowd density estimation on various crowd datasets, demonstrating a wide range of applications for real-time monitoring in public crowd management.

The Performance of the Optical Flow Field based Dense Image Matching for UAV Imagery

Abstract. With the rapid development of sensing platforms, unmanned aerial vehicle (UAV)-based mapping has become increasingly popular because of its economic efficiency and flexibility, especially for providing 3D information to support urban growth monitoring and change detection to meet sustainable development goals (SDGs). This paper presents an improved optical flow field-based dense matching algorithm (OFFDM) for low-altitude UAV images based on the Ph.D. thesis of Yuan (Yuan, 2018). First, high-precision seed points were used to compute the optical flow field within stereo pairs, effectively minimizing redundant calculations during the fine-matching phase. Second, a fine-matching approach, integrating multiple constraints, was applied to refine the coarse matching results based on the optical flow field. Extensive dense matching experiments on UAV low-altitude aerial imagery assessed the performance of OFFDIM across four dimensions: 3D point cloud visualization, matching success rate, precision, and reliability. Extensive experiments on low-altitude UAV imagery, characterized by a resolution of 7cm per pixel over a 10,608×8,608 pixel dimension and a 60% forward overlap, evaluate the OFFDM's efficacy. The quantitative evaluation revealed that the proposed method achieved an accuracy of ±0.7 pixels in image coordinates and ±20 cm on the ground, with a matching success rate exceeding 97%. The processing time was approximately 272 seconds for handling one single stereo pair. When compared to the widely adopted PMVS algorithm, known for its effectiveness in dense matching for UAV images, the proposed method demonstrated higher completeness and improved matching efficiency by more than five times. These results demonstrated that the proposed approach is more suitable for dense matching on UAV imagery-based high-precision 3D spatial data extraction, supporting global mapping tasks more effectively.

Ghost cells as a two-phase blood analog fluid-high-volume and high-concentration production.

Hemolysis in mechanical circulatory support systems is currently determined quantitatively. To also locally resolve hemolysis, we are developing a fluorescent hemolysis detection method. This requires a translucent two-phase blood analog fluid combined with particle image velocimetry, an optical flow field measurement. The blood analog fluid is composed of red blood cell surrogates. However, producing surrogates in sufficient volume is a challenge. We therefore present a high-volume and high-concentration production for our surrogates: ghost cells, hemoglobin-depleted erythrocytes. In the ghost cell production, the hemoglobin is removed by a repeated controlled osmolar lysis. We have varied the solution mixture, centrifugation time, and centrifugation force in order to increase production efficiency. The production is characterized by measurements of output volume, hematocrit, transparency, and rheology of the blood analog fluid. The volume of produced ghost cells was significantly increased, and reproducibility was improved. An average production of 389 mL of ghost cells were achieved per day. Those ghost cells diluted in plasma have a rheology similar to blood while being permeable to light. The volume of ghost cells produced is sufficient for optical measurements as particle image velocimetry in mechanical circulatory support systems. This makes further work on experimental measurements for a locally resolved hemolysis detection possible.

Laboratory Abnormal Behavior Detection Based on Multimodal Information Fusion

The traditional laboratory anomaly detection methods mainly focus on the hidden dangers caused by chemical leaks and other items, ignoring the impact of abnormal behaviors such as incorrect operations and improper behavior on safety in the laboratory. This paper proposes a laboratory abnormal behavior detection method based on multimodal information fusion. The method generates a dense optical flow field of RGB image sequences based on optical flow theory and global smoothing constraints, and mines motion mode information. Meanwhile, the contour modal information of behavior is captured through convolution and adjacency matrix operations. Using decision level and proximity functions to integrate student behavior motion mode information and contour mode information, and using the maximum value as the behavior detection result. The experimental results show that the method can effectively detect abnormal behavior in the laboratory environment, with small detection errors and a specificity close to 1.00, effectively ensuring the safety of the laboratory environment.

Enhanced river suspended sediment concentration identification via multimodal video image, optical flow, and water temperature data fusion

Monitoring suspended sediment concentration (SSC) in rivers is pivotal for water quality management and sustainable river ecosystem development. However, achieving continuous and precise SSC monitoring is fraught with challenges, including low automation, lengthy measurement processes, and high cost. This study proposes an innovative approach for SSC identification in rivers using multimodal data fusion. We developed a robust model by harnessing colour features from video images, motion characteristics from the Lucas–Kanade (LK) optical flow method, and temperature data. By integrating ResNet with a mixed density network (MDN), our method fused the image and optical flow fields, and temperature data to enhance accuracy and reliability. Validated at a hydropower station in the Xinjiang Uygur Autonomous Region, China, the results demonstrated that while the image field alone offers a baseline level of SSC identification, it experiences local errors under specific conditions. The incorporation of optical flow and water temperature information enhanced model robustness, particularly when coupling the image and optical flow fields, yielding a Nash–Sutcliffe efficiency (NSE) of 0.91. Further enhancement was observed with the combined use of all three data types, attaining an NSE of 0.93. This integrated approach offers a more accurate SSC identification solution, enabling non-contact, low-cost measurements, facilitating remote online monitoring, and supporting water resource management and river water–sediment element monitoring.

SFD-SLAM: a novel dynamic RGB-D SLAM based on saliency region detection

Abstract In dynamic environments, several simultaneous localization and mapping (SLAM) systems effectively utilize optical flow fields to distinguish dynamic from static feature points. Commonly, these systems leverage the amplitude information within the optical flow field to develop adaptive thresholding segmentation models for identifying dynamic scene regions. Nevertheless, designing adaptive thresholding models typically necessitates meticulous planning and extensive experimentation. This study introduces a dynamic RGBD SLAM system, SFD-SLAM, which innovates by employing a saliency detection network for the direct extraction of dynamic regions via scene flow. This approach notably streamlines the design process associated with conventional adaptive thresholding models. Furthermore, SFD-SLAM incorporates a geometric module that merges depth residuals with hyperpixel segmentation to enhance the refinement of the dynamic mask. This is followed by integration with FCM clustering for the precise identification of moving objects. The efficacy of SFD-SLAM is assessed using the widely recognized TUM dynamic dataset. Experimental results demonstrate that the proposed system surpasses DGFlow-SLAM, which relies on an adaptive thresholding model for dynamic object segmentation, in terms of trajectory accuracy. It also achieves comparable localization accuracy to DynaSLAM. Moreover, SFD-SLAM maintains robust tracking capabilities, even in scenarios where DynaSLAM experiences tracking loss, thereby augmenting the robustness of RGBD-SLAM in dynamic settings.

A deep learning workflow enhanced with optical flow fields for flood risk estimation

A deep learning workflow enhanced with optical flow fields for flood risk estimation

Feature Scalar Field Grid-Guided Optical-Flow Image Matching for Multi-View Images of Asteroid

Images captured by deep space probes exhibit large-scale variations, irregular overlap, and remarkable differences in field of view. These issues present considerable challenges for the registration of multi-view asteroid sensor images. To obtain accurate, dense, and reliable matching results of homonymous points in asteroid images, this paper proposes a new scale-invariant feature matching and displacement scalar field-guided optical-flow-tracking method. The method initially uses scale-invariant feature matching to obtain the geometric correspondence between two images. Subsequently, scalar fields of coordinate differences in the x and y directions are constructed based on this correspondence. Next, interim images are generated using the scalar field grid. Finally, optical-flow tracking is performed based on these interim images. Additionally, to ensure the reliability of the matching results, this paper introduces three methods for eliminating mismatched points: bidirectional optical-flow tracking, vector field consensus, and epipolar geometry constraints. Experimental results demonstrate that the proposed method achieves a 98% matching correctness rate and a root mean square error of 0.25 pixels. By combining the advantages of feature matching and optical-flow field methods, this approach achieves image homonymous point matching results with precision and density. The matching method exhibits robustness and strong applicability for asteroid images with cross-scale, large displacement, and large rotation angles.

The Influence of Substrate on the Optical Properties of Gold Nanoslits.

Nanoslits have various applications, including localized surface plasmon resonance (LSPR)-based nanodevices, optical biosensors, superfocusing, high-efficiency refractive index sensors and chip-based protein detection. In this study, the effect of substrates on the optical properties of gold nanoslits placed in free space is discussed; for this purpose, glass BK7 and Al2O3 are used as substrates and the wavelength of incident light is supposed to be 650 nm. The optical properties, power flow and electric field enhancement for gold nanoslits are investigated by using the finite element method (FEM) in COMSOL Multiphysics software. The effect of polarization of an incident electromagnetic wave as it propagates from a gold nanoslit is also analyzed. As special case, the effect of glass and alumina substrate on magnetic field, power flow and electric field enhancement is discussed. The goal of this research is to investigate the phenomenon of power flow and electric field enhancement. The study of power flow in gold nanoslits provides valuable insights into the behavior of light at the nanoscale and offers opportunities for developing novel applications in the field of nanophotonics and plasmonics. The consequences of this study show the significance of gold nanoslits as optical nanosensors.

Random epipolar constraint loss functions for supervised optical flow estimation

The majority of supervised models estimate optical flow through minimizing the numerical difference between the predicted flow and the ground truth, resulting in the loss of positional and geometric characteristics of the calculated flow fields. In addition, these models require a large number of parameters and high computational cost when computing optical flow. To address these issues, this paper presents a novel loss function and a lightweight framework for optical flow estimation. The proposed loss function, called the random epipolar constraint loss function (RECLoss), incorporates epipolar geometry into supervised optimization to transform the numerical difference into geometry constraint. The RECLoss can make the optical flow estimation models more effective and enhance their generalization abilities. Moreover, the design of RECLoss is more interpretable and the estimated optical flow fields from RECLoss have clearly defined mathematical meanings. A lightweight recurrent neural network for optical flow estimation (LRFlow) that balances computational cost and estimation accuracy is also proposed. The LRFlow, containing only 3.0M parameters, consists of a feature extractor, a correlation matching module, and an iterative update unit. The proposed lightweight network achieves state-of-the-art results compared to all other lightweight networks on the challenging MPI-Sintel and KITTI2015 datasets. The effectiveness of RECLoss in improving the accuracy of LRFlow and other state-of-the-art methods such as RAFT and GMA has been validated through extensive experiments. The source code of the project are available at https://github.com/Eryo-iPython/RECLoss.

Anomalous binocular vision in African Harrier-Hawks

An animal's visual field is the three-dimensional space around its head from which it can extract visual information at any instant1. Bird visual fields vary markedly between species, and this variation is likely to be driven primarily by foraging ecology1,2,3. The binocular visual field is the region in which the visual fields of the two eyes overlap; thus, objects in the binocular field are imaged by both eyes simultaneously. The binocular field plays a pivotal role in the detection of symmetrical optic flow-fields, providing almost instantaneous information on the direction of travel and the time to contact a target towards which the head or feet is travelling; thus, information from the binocular field is crucial in guiding key foraging behaviours2,3. Here, we demonstrate an unusual visual field and binocular extent above the head in African Harrier-Hawks, also known as Gymnogenes (Polyboroides typus) compared to 18 other members of the Accipitridae4,5. We argue that the observed visual field can be attributed to the unusual and specific foraging behaviour of African Harrier-Hawks.

Atmospheric Turbulence Degraded Video Restoration with Recurrent GAN (ATVR-GAN).

Atmospheric turbulence (AT) can change the path and direction of light during video capturing of a target in space due to the random motion of the turbulent medium, a phenomenon that is most noticeable when shooting videos at long ranges, resulting in severe video dynamic distortion and blur. To mitigate geometric distortion and reduce spatially and temporally varying blur, we propose a novel Atmospheric Turbulence Video Restoration Generative Adversarial Network (ATVR-GAN) with a specialized Recurrent Neural Network (RNN) generator, which is trained to predict the scene's turbulent optical flow (OF) field and utilizes a recurrent structure to catch both spatial and temporal dependencies. The new architecture is trained using a newly combined loss function that counts for the spatiotemporal distortions, specifically tailored to the AT problem. Our network was tested on synthetic and real imaging data and compared against leading algorithms in the field of AT mitigation and image restoration. The proposed method outperformed these methods for both synthetic and real data examined.

Enhancing Rainfall Nowcasting Using Generative Deep Learning Model with Multi-Temporal Optical Flow

Precipitation nowcasting is critical for preventing damage to human life and the economy. Radar echo tracking methods such as optical flow algorithms have been widely employed for precipitation nowcasting because they can track precipitation motions well. Thus, this method, including the McGill algorithm for precipitation nowcasting by Lagrangian extrapolation (MAPLE), was implemented for operational precipitation nowcasting. However, advection-based methods struggle to predict the nonlinear motions of precipitation fields and dynamic processes, such as the growth and decay of precipitation. This study proposes an enhanced optical flow model using a multi-temporal optical flow field and a conditional generative adversarial network (cGAN). We trained the proposed model using a 3-year radar dataset provided by the Korean Meteorological Administration and performed forecast skill evaluations using both qualitative and quantitative methods. In particular, the model featuring multi-temporal optical flow enhances prediction accuracy for the nonlinear motion of precipitation fields, and the model’s accuracy can be further improved through the use of the cGAN structure. We have verified that these improvements hold for 0–3 h lead times. Based on this performance enhancement, we conclude that the multi-temporal optical flow model with cGAN has a potential role in operational precipitation nowcasting.

A Highly Compressed Accelerator With Temporal Optical Flow Feature Fusion and Tensorized LSTM for Video Action Recognition on Terminal Device

Deep learning based action recognition has become ubiquitous in the video analysis area; however, large neural networks require enormous computations to achieve high performance, which hinder them from mobile applications that are tightly constrained by hardware resources. In this work, we introduce a highly compact and fast neural network based Action Recognition Accelerator named ARA on the terminal device. We build an LSTM based spatio-temporal action recognition model with extracted time-series features from RGB frames and flow features from optical flow fields. Then the LSTM based spatio-temporal model is deeply compressed with tensor decomposition to further reduce redundant parameters and lessen computation overhead. Based on the datasets UCF-11, UCF-101, and HMDB51, our proposed method achieves 95.87%, 94.08%, and 75.71% classification accuracy, being comparable with other state-of-the-art methods. In particular, our proposed method significantly compresses the parameter of the LSTM model 215× on the UCF-101 dataset. The proposed system can also achieve a fast running speed of 157.7 FPS on GPU. Furthermore, we validate the performance of the proposed system on an ARM-based terminal device; the results show it only has 0.017s latency and 4.73W power consumption.

Nodes2STRNet for structural dense displacement recognition by deformable mesh model and motion representation

AbstractDisplacement is a critical indicator for mechanical systems and civil structures. Conventional vision‐based displacement recognition methods mainly focus on the sparse identification of limited measurement points, and the motion representation of an entire structure is very challenging. This study proposes a novel Nodes2STRNet for structural dense displacement recognition using a handful of structural control nodes based on a deformable structural three‐dimensional mesh model, which consists of control node estimation subnetwork (NodesEstimate) and pose parameter recognition subnetwork (Nodes2PoseNet). NodesEstimate calculates the dense optical flow field based on FlowNet 2.0 and generates structural control node coordinates. Nodes2PoseNet uses structural control node coordinates as input and regresses structural pose parameters by a multilayer perceptron. A self‐supervised learning strategy is designed with a mean square error loss and L2 regularization to train Nodes2PoseNet. The effectiveness and accuracy of dense displacement recognition and robustness to light condition variations are validated by seismic shaking table tests of a four‐story‐building model. Comparative studies with image‐segmentation‐based Structure‐PoseNet show that the proposed Nodes2STRNet can achieve higher accuracy and better robustness against light condition variations. In addition, NodesEstimate does not require retraining when faced with new scenarios, and Nodes2PoseNet has high self‐supervised training efficiency with only a few control nodes instead of fully supervised pixel‐level segmentation.

A Robust Semi-Direct 3D SLAM for Mobile Robot Based on Dense Optical Flow in Dynamic Scenes.

Dynamic objects bring about a large number of error accumulations in pose estimation of mobile robots in dynamic scenes, and result in the failure to build a map that is consistent with the surrounding environment. Along these lines, this paper presents a robust semi-direct 3D simultaneous localization and mapping (SLAM) algorithm for mobile robots based on dense optical flow. First, a preliminary estimation of the robot's pose is conducted using the sparse direct method and the homography matrix is utilized to compensate for the current frame image to reduce the image deformation caused by rotation during the robot's motion. Then, by calculating the dense optical flow field of two adjacent frames and segmenting the dynamic region in the scene based on the dynamic threshold, the local map points projected within the dynamic regions are eliminated. On this basis, the robot's pose is optimized by minimizing the reprojection error. Moreover, a high-performance keyframe selection strategy is developed, and keyframes are inserted when the robot's pose is successfully tracked. Meanwhile, feature points are extracted and matched to the keyframes for subsequent optimization and mapping. Considering that the direct method is subject to tracking failure in practical application scenarios, the feature points and map points of keyframes are employed in robot relocation. Finally, all keyframes and map points are used as optimization variables for global bundle adjustment (BA) optimization, so as to construct a globally consistent 3D dense octree map. A series of simulations and experiments demonstrate the superior performance of the proposed algorithm.

A Multi-Modal Transformer network for action detection

This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera movement. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative experiments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools.

Transverse Velocity Field Measurements in High-resolution Solar Images Based on Deep Learning

To address the problem of the low accuracy of transverse velocity field measurements for small targets in high-resolution solar images, we proposed a novel velocity field measurement method for high-resolution solar images based on PWCNet. This method transforms the transverse velocity field measurements into an optical flow field prediction problem. We evaluated the performance of the proposed method using the Hα and TiO data sets obtained from New Vacuum Solar Telescope observations. The experimental results show that our method effectively predicts the optical flow of small targets in images compared with several typical machine- and deep-learning methods. On the Hα data set, the proposed method improves the image structure similarity from 0.9182 to 0.9587 and reduces the mean of residuals from 24.9931 to 15.2818; on the TiO data set, the proposed method improves the image structure similarity from 0.9289 to 0.9628 and reduces the mean of residuals from 25.9908 to 17.0194. The optical flow predicted using the proposed method can provide accurate data for the atmospheric motion information of solar images. The code implementing the proposed method is available on https://github.com/lygmsy123/transverse-velocity-field-measurement.

Motion Estimation for Complex Fluid Flows Using Helmholtz Decomposition

In this paper, we proposed a novel motion model with Helmholtz decomposition for complex fluid flows in a filtering-based optical flow framework, where the optimization of the regularization term is treated as a filtering process, and different motion patterns can be captured by finding appropriate filter kernels based on the designed filters. In this framework, we introduce a novel optical flow method with a joint spatial filter, which is based on the Helmholtz decomposition theorem that assumes a local motion field is composed of a curl field and a divergence field. By adjusting the scale of the weights in the filter kernels and combining a curl filter with a divergence filter in a certain ratio, it can simulate different motion patterns. In addition, if the correlation between the horizontal and vertical components of the optical flow field in the filter kernel is eliminated, it will be transformed into a linear motion model. Based on this linear motion model, we also develop a novel optical flow method with an adaptive guided filter. By finding an adaptive filter kernel driven by both the input image and the guided motion field for the designed filter, it can successfully capture different motion patterns, and yield an edge-preserving smoothing optical flow field. Most importantly, the proposed optical flow model provides a new way to design the regularizers for capturing different motion patterns in complex fluid flow. In particular, the designed optical flow method with an adaptive guided filter significantly outperforms the current state-of-the-art optical flow methods in predicting complex fluid flows.