Local Optical Flow Research Articles

Maritime Infrared Small Target Detection Based on the Appearance Stable Isotropy Measure in Heavy Sea Clutter Environments.

Infrared small target detection plays a crucial role in maritime security. However, detecting small targets within heavy sea clutter environments remains challenging. Existing methods often fail to deliver satisfactory performance in the presence of substantial clutter interference. This paper analyzes the spatial-temporal appearance characteristics of small targets and sea clutter. Based on this analysis, we propose a novel detection method based on the appearance stable isotropy measure (ASIM). First, the original images are processed using the Top-Hat transformation to obtain the salient regions. Next, a preliminary threshold operation is employed to extract the candidate targets from these salient regions, forming a candidate target array image. Third, to distinguish between small targets and sea clutter, we introduce two characteristics: the gradient histogram equalization measure (GHEM) and the local optical flow consistency measure (LOFCM). GHEM evaluates the isotropy of the candidate targets by examining their gradient histogram equalization, while LOFCM assesses their appearance stability based on local optical flow consistency. To effectively combine the complementary information provided by GHEM and LOFCM, we propose ASIM as a fusion characteristic, which can effectively enhance the real target. Finally, a threshold operation is applied to determine the final targets. Experimental results demonstrate that our proposed method exhibits superior comprehensive performance compared to baseline methods.

A local optical flow eye-tracking method for depression detection

Aiming at the problem that depression detection is easily affected by subjective factors such as patient willingness, self-awareness, and doctor’s level, as well as inconvenient detection and high cost, a universal detection method with low cost and convenient deployment is proposed. A local optical flow tracking method is adopted, which is realized by eye region detection and optical flow estimation. Then, three experiments are designed to verify the accuracy of the method and the effectiveness of eye movement tracking. The formula for calculating the depression state is proposed, and the eye movement of one normal participant’s elliptical trajectory of the left and right pupils in the horizontal and vertical directions is 44.4805, 42.7013, 29.3571, and 31.1364 mm, respectively. To verify the effectiveness of the proposed method, three example points are selected. Among them, the fixation duration at P1 is 1.87 s, the eye movement direction at P2 and P3 is 1.168°, counterclockwise around the X axis, and the number of fixation times is 46. Finally, 18 normal participants and four depression-positive participants are tested by the exploratory eye movement paradigm. The results show that the accuracy rate of identifying depression patients using the NEF metric method can reach ∼81.8%, which can help quantify the detection of depression and provide the possibility for professionals to remotely diagnose individuals with depression.

Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint

INTRODUCTION: Video inpainting is a very important task in computer vision, and it’s a key component of various practical applications. It also plays an important role in video occlusion removal, traffic monitoring and old movie restoration technology. Video inpainting is to obtain reasonable content from the video sequence to fill the missing region, and maintain time continuity and spatial consistency.OBJECTIVES: In previous studies, due to the complexity of the scene of video inpainting, there are often cases of fast motion of objects in the video or motion of background objects, which will lead to optical flow failure. So the current video inpainting algorithm hasn’t met the requirements of practical applications. In order to avoid the problem of optical flow failure, this paper proposes a transformer-guided video inpainting model based on local Spatial-temporal joint.METHODS: First, considering the rich Spatial-temporal relationship between local flows, a Local Spatial-Temporal Joint Network (LSTN) including encoder, decoder and transformer module is designed to roughly inpaint the local corrupted frames, and the Deep Flow Network is used to calculate the local bidirectional corrupted flows. Then, the local corrupted optical flow map is input into the Local Flow Completion Network (LFCN) with pseudo 3D convolution and attention mechanism to obtain a complete set of bidirectional local optical flow maps. Finally, the roughly inpainted local frame and the complete bidirectional local optical flow map are sent to the Spatial-temporal transformer and the inpainted video frame is output.RESULTS: Experiments show that the algorithm achieves high quality results in the video target removal task, and has a certain improvement in indicators compared with advanced technologies.CONCLUSION: Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint can obtain high-quality optical flow information and inpainted result video.

Study of Global Photospheric and Chromospheric Flows Using Local Correlation Tracking and Machine Learning Methods I: Methodology and Uncertainty Estimates

Cyclical variations of the solar magnetic fields, and hence the level of solar activity, are among the top interests of space weather research. Surface flows in global-scale, in particular differential rotation and meridional flows, play important roles in the solar dynamo that describes the origin and variation of solar magnetic fields. In principle, differential rotation is the fundamental cause of dipole field formation and emergence, and meridional flows are the surface component of a longitudinal circulation that brings decayed field from low latitudes to polar regions. Such flows are key inputs and constraints of observational and modeling studies of solar cycles. Here, we present two methods, local correlation tracking (LCT) and machine learning-based self-supervised optical flow methods, to measure differential rotation and meridional flows from full-disk magnetograms that probe the photosphere and text{H}alpha images that probe the chromosphere, respectively. LCT is robust in deriving photospheric flows using magnetograms. However, we found that it failed to trace flows using time-sequence text{H}alpha data because of the strong dynamics of traceable features. The optical flow methods handle text{H}alpha data better to measure the chromospheric flow fields. We found that the differential rotation from photospheric and chromospheric measurements shows a strong correlation with a maximum of 2.85~upmu text{rad},text{s}^{-1} at the equator and the accuracy holds until 60^{circ } for the MDI and text{H}alpha, 75^{circ } for the HMI dataset. On the other hand, the meridional flow deduced from the chromospheric measurement shows a similar trend as the concurrent photospheric measurement within 60^{circ } with a maximum of 20~text{m},text{s}^{-1} at 40^{circ } in latitude. Furthermore, the measurement uncertainties are discussed.

Distance estimation in the goldfish (Carassius auratus).

Neurophysiological advances have given us exciting insights into the systems responsible for spatial mapping in mammals. However, we are still lacking information on the evolution of these systems and whether the underlying mechanisms identified are universal across phyla, or specific to the species studied. Here we address these questions by exploring whether a species that is evolutionarily distant from mammals can perform a task central to mammalian spatial mapping–distance estimation. We developed a behavioural paradigm allowing us to test whether goldfish (Carassius auratus) can estimate distance and explored the behavioural mechanisms that underpin this ability. Fish were trained to swim a set distance within a narrow tank covered with a striped pattern. After changing the background pattern, we found that goldfish use the spatial frequency of their visual environment to estimate distance, doubling the spatial frequency of the background pattern resulted in a large overestimation of the swimming distance. We present robust evidence that goldfish can accurately estimate distance and show that they use local optic flow to do so. These results provide a compelling basis to use goldfish as a model system to interrogate the evolution of the mechanisms that underpin spatial cognition, from brain to behaviour.

Endoscopic OCT Angiography Using Clinical Proximal-End Scanning Catheters

Endoscopic optical coherence tomography angiography (OCTA) is a promising modality to inspect the microvasculature of inner organs in the early-stage tumor diagnosis. However, an endoscopic clinical proximal-end scanning catheter has limited flow imaging capability due to the nonuniform rotational distortion (NURD) and physiological motion. In this study, a combined local and global (CLG) optical flow algorithm was used to estimate the motion vectors caused by NURD and physiological motion. The motion vectors were used to bicubic-interpolation-resample the OCT structure to ensure that the circumferential pixels were equally spaced in the space domain. Then, angiograms were computed based on the statistical relation between inverse SNR (iSNR) and amplitude decorrelation (IDa), termed as IDa-OCTA. Finally, the ability of this technique for endoscopic OCTA imaging was demonstrated by flow phantom experiments and human nailfold capillary imaging.

Effectiveness Of Moving Objects Detecting And Tracking In Airspace By Images In NearInfrared

Objects in nearinfrared (NIR) images can often have different linear scales and shapes than the same objects in optical images for visible spectrum (RedGreenBlue, RGB). Therefore they can require different computer vision methods for detection, tracking, and classification. This paper devoted to the methods by which the problems of moving objects detecting and tracking in NIR images are solved. The main characteristics of moving objects in image sequences are highlighted. Advantages and disadvantages of different methods for detecting and tracking objects in NIR images of airspace are considered and two of the most promising methods classes are selected. Studies have been carried out on the effectiveness of LucasKanade method, which is one of the methods of local optical flow, and the ORB method of scaleinvariant transformation of features when detecting and tracking moving objects in NIR images. In numerical experiments, more than 5000 NIR images containing moving objects of three types were used as well as three combinations of considered methods. It is shown which combination is the most accurate in the tasks of moving objects detection and tracking and can be used for airspace automatic operational control and management based on computer vision systems. Probably, other combinations of methods from the two considered classes also can help to increase accuracy of moving objects detection and tracking in airspace by NIR images.

UniVIO: Unified Direct and Feature-Based Underwater Stereo Visual-Inertial Odometry

Providing an accurate trajectory in an underwater scene is challenging for visual odometry due to weak texture and varying illumination. In this article, we propose a novel underwater visual-inertial odometry (VIO) with the unifying direct method and feature-based method. Our approach starts with a new rectification method for underwater images. The new underwater image rectification method separately eliminates water-air refraction distortion and lens distortion with an approximate single viewpoint (SVP) camera model. Meanwhile, a unified optimization method is used in our approach to obtain robust and highly accurate odometry results. This method jointly optimizes projection errors and photometric errors. Moreover, the robust data association processing, which combines <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keyframe</i> / <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> selection, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> position prediction, and robust local optical flow, was designed in our approach. The robust data association processing greatly improves the success rate of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoints</i> tracking. To evaluate the precision of underwater visual odometry, an underwater dataset with complete and highly accurate ground truth trajectories is provided in this work. The performance of our system is compared with other state-of-the-art algorithms on multiple datasets, including visual odometry datasets and our proposed underwater dataset. The evaluation results show that the proposed system can achieve better results than the state-of-the-art algorithms. Furthermore, the proposed system is validated on public natural underwater datasets to show the performance of the system in a real underwater environment. The qualitative results demonstrate that the proposed system can work well in the natural underwater environment.

Adjustable window for 2D PIV estimation based on local Lagrangian coherency

We present a novel approach to adjust shapes of the interrogation windows (IW) in Particle Image Velocimetry (PIV) measurements as a function of temporal and spatial local coherent motions. Lagrangian Coherent Structures (LCS) has been widely utilized to determine local flow boundaries. We propose using Finite-Time Lyapunov Exponent (FTLE) to quantify LCS separatrix boundaries (i.e. ridges) and adjust the interrogation window. We integrated the proposed method with a local optical flow PIV algorithm. The evaluation was performed using synthetic particle images of 2D homogeneous isotropic turbulence obtained from Direct Numerical Simulation (DNS). The results showed significant improvements in regions with complex flow behaviours, particularly shear, vortex and hyperbolic motions. We studied improvements of the velocity estimation in a real experiment of the wake flow behind a cylinder at Reynolds number equal to 3900. It was found that optical flow featured by coherency based interrogation window (coherent optical flow) reveals detailed vector field estimations in regions with complex behaviours inside the wake flow.

An Improved Fractional-Order Variational Optical Flow Model Combining Structure Tensor

Dealing with problems of illumination changes in optical flow estimation, an improved variational optical flow model is proposed in this paper. The local structure constancy constraint (LSCC) is applied in the data term of the traditional HS (Horn &amp; Schunck) optical flow model to substitute the brightness constancy constraint. The fractional-order smoothness constraint (FSC) is applied in the smoothness term of the HS model. Then, the detailed calculation processes from the optical flow model to the optical flow value are explained. The structure tensor in LSCC is an image feature that is constant in the illumination changes scene. The fractional differential coefficient in FSC can fuse the local neighborhood optical flow vector into the optical flow vector of the target pixel, which can improve the integrity of the motion region with the same motion speed. Combining LSCC with FSC, our improved optical flow model can obtain an accurate optical flow field with clear outline in the illumination abnormity scene. The experimental results show that, compared with other optical flow models, our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.

Research on traffic flow detection algorithm based on road video

When using the road monitoring video to detect the traffic flow, the vehicle may be treated as one due to the close distance in the video, or the vehicle may be misdetected or missed due to noise interference, resulting in inaccurate traffic data. In order to solve this problem, this paper proposes to generate a foreground for road vehicles by LK local differential optical flow method, calculates the Euclidean distance between the vehicle and the background by using the base watershed algorithm, screens the vehicle position, and segment the adhesion vehicles. At the same time, according to the best position of the vehicle in the video, the virtual detection line is set. The Kalman filter is used to optimise the monitoring data, the traffic flow data statistics error problem is solved, and the accurate traffic flow data is obtained. The experimental result shows that the method mentioned in the paper can obtain accurate traffic data.

Study on Local Optical Flow Method Based on YOLOv3 in Human Behavior Recognition

In the process of human behavior recognition, the traditional dense optical flow method has too many pixels and too much overhead, which limits the running speed. This paper proposed a method combing YOLOv3 (You Only Look Once v3) and local optical flow method. Based on the dense optical flow method, the optical flow modulus of the area where the human target is detected is calculated to reduce the amount of computation and save the cost in terms of time. And then, a threshold value is set to complete the human behavior identification. Through design algorithm, experimental verification and other steps, the walking, running and falling state of human body in real life indoor sports video was identified. Experimental results show that this algorithm is more advantageous for jogging behavior recognition.

A fast human action recognition network based on spatio-temporal features

Artificial intelligence models are widely used in the field of human activity recognition, and human action recognition is an important aspect of human activity recognition. The core of human action recognition is to understand the temporal relationship between video frames. Almost all state-of-the-art methods of human action recognition in videos use optical flow. However, traditional local optical flow estimation methods areexpensive and not trained end-to-end. In this paper, we propose a fast network for human action recognition. Our purpose is to improve the efficiency of optical flow feature extraction and explore the fusion method of spatio-temporal features. For spatio-temporal features, our method combines spatial features and temporal features into fusion features. In addition, we propose CNN with OFF instead of the VGG16 network, which is used to process optical flow features to obtain abundant features. Our model only needs RGB inputs to get the state-of-the-art accuracy of 91.5% on UCF-101, 67.9% on HMDB51, 83.3% on MSR Daily Activity3D, and 91.25% on Florence 3D action, respectively. Compared with most state-of-the-art video action recognition models, our proposed model can effectively improve the accuracy of human action recognition.

Video frame interpolation via optical flow estimation with image inpainting

As we all know, video frame rate determines the quality of the video. The higher the frame rate, the smoother the movements in the picture, the clearer the information expressed, and the better the viewing experience for people. Video interpolation aims to increase the video frame rate by generating a new frame image using the relevant information between two consecutive frames, which is essential in the field of computer vision. The traditional motion compensation interpolation method will cause holes and overlaps in the reconstructed frame, and is easily affected by the quality of optical flow. Therefore, this paper proposes a video frame interpolation method via optical flow estimation with image inpainting. First, the optical flow between the input frames is estimated via combined local and global-total variation (CLG-TV) optical flow estimation model. Then, the intermediate frames are synthesized under the guidance of the optical flow. Finally, the nonlocal self-similarity between the video frames is used to solve the optimization problem, to fix the pixel loss area in the interpolated frame. Quantitative and qualitative experimental results show that this method can effectively improve the quality of optical flow estimation, generate realistic and smooth video frames, and effectively increase the video frame rate.

METHOD OF OPTICAL FLOW CALCULATION BASED ON FEATURE AUGMENTATION

The paper addresses the problem of robust optical flow computation for real-time applications. We present an efficient, in presence of varying illumination and large displacements, method of dense optical flow calculation based on Robust Local Optical Flow (RLOF). Introduction of Adaptive and Generic Accelerated Segment Test (AGAST) feature detector with additional augmentation by simple grid points provides a stable and reliable under difficult conditions set of features and uniform points that produces dense and accurate optical flow and leads to better performance. The usage of feature points for prior initialization leads for robustness for large displacements as well. The other essential step of optical flow calculation – densification of optical flow – was strengthen by modification of weights. The raster interpolation was applied with weights on second derivatives to make more edgesensitive optical flow. To enhance the overall result, a post-processing step of variational refinement process were added to optical flow calculation framework. The proposed method was evaluated on Max Planck Institute (MPI) Sintel dataset – a challenging set of 3D-animated movies with large range of motions and illuminations, which has ground truth optical flow available. The comparison with initial RLOF method were measured by running time and quality parameters, measured by End Point Error (EPE) and Angular Error (AE). The evaluation shows the decrease of running time by approximately 40 % and improvement of EPE and AE, that demonstrates the effectiveness of the method.

Low dose 4D-CT super-resolution reconstruction via inter-plane motion estimation based on optical flow

In order to obtain the dynamic intuitive image of the patient's internal organs movement and minimize the potential risks of X-ray radiation at the same time, low dose four-dimensional computed tomography (4D-CT) has attracted a considerable interest in the high precision radiation therapy. But some susceptible artifacts, including device-dependent, image reconstruction times and patient's respiratory pattern, usually cause an inter-plane thickness that is much greater than intra-plane voxel resolutions. In this study, to estimate the respiratory motion and enhance the inter-plane resolution of multi-plane computed tomography (CT) images, a joint optimization framework was proposed using the combined local and global (CLG) variational optical flow and improved non-local iterative back projection (NLIBP). Note that, the premise of this work is that the anatomical information missing in one particular phase can be recovered from other phases in CT images. First, CLG variational optical flow model was constructed to estimate the respiratory motion (i.e., the optical flow fields) between different phases at the corresponding voxel positions, and then was solved by the fast alternating direction method of multipliers (ADMM). Secondly, the improved NLIBP algorithm characterized by non-local mean filter and image fusion strategy was employed to reconstruct high resolution (HR) inter-plane images based on the calculated motion fields. Finally, we explored different hyperparameter settings to achieve a good trade-off between the super resolution (SR) reconstruction performance and computational efficiency, and indicated the success of CLG variational optical flow method for estimating the displacement field between images. Experimental results on public lung 4D-CT datasets demonstrated that this proposed method is able to more effectively enhance texture structures while preserving edges, and outperforms current state-of-the-art methods both quantitatively and qualitatively.

Non-local weighted regularization for optical flow estimation

Global optical flow characterizes the overall motion while local optical flow characterizes the individual's movement. Therefore, both global and local optical flow is important to analyze the motion of each object in an image and of the camera. In order to robustly and accurately estimate the optical flow which includes both global and local motion information, this paper proposes a novel variational framework with non-local weighted regularization for optical flow estimation (NLWOF). The proposed NLWOF strategy includes the following two key parts: Firstly, non-local prior is used a regularization to robustly estimate the local optical flow due to that the similar structure patches using in non-local weight can suppress noise interference. Secondly, the solution of the NLWOF model can be approximated by the employment of the Lagrange–Euler formula and the approximation of the Laplace operator. Experimental results in both quantitation and qualitation verify the effectiveness of the NLWOF scheme, and even better than those of the state-of-the-arts.

Evaluating a potential technique with local optical flow vectors for automatic organ-at-risk (OAR) intrusion detection and avoidance during radiotherapy

Various techniques of deep inspiration breath hold (DIBH) have been used to mitigate the likelihood and risk of exposing the heart, an organ-at-risk (OAR) for unintended radiation during left breast radiotherapy. However, issues of reproducibility of these techniques warrant further investigation into the feasibility of detecting the intrusion of an OAR into the treatment field during intra-fractional treatment delivery. The increase of high-dose, low-fraction radiotherapy treatments makes it important to immediately adapt treatment once an OAR is detected in the treatment field. This proof-of-concept implementation includes an algorithm that detects and tracks the motion at the edges of a treatment field and a control algorithm that adapts the treatment aperture according to the motion detected. In accordance to the AAPM Task-Group (TG-132) report, image registration techniques should be verified with virtual and physical phantoms prior to clinical application. Since most OARs move as a result of respiration-induced motion, we have used a lung phantom to generate images of a generic OAR intruding into a treatment field with known velocity. The phantom was programmed to move with sinusoidal and lung patient tumor motion patterns and the accuracy of intrusion tracking and MLC adaptation were benchmarked with the ground truth—programmed motion of the OAR. The motions were recorded with an electronic portal imaging device (EPID). An optimal cluster size of 9 × 9 motion vectors was found to provide the smallest average absolute position error of 0.3 mm. A strong linear correlation between the adapted MLC leaves and the actual OAR position was observed. The algorithm had a mean position tracking error of −0.4 ± 0.3 mm and a precision of 1.1 mm. It is possible to adapt MLC leaves based on the motion detected at the edges of the irradiated field, and it would be feasible to shield an unplanned intrusion of an OAR into the treatment field.

A Global Method for Non-Rigid Registration of Cell Nuclei in Live Cell Time-Lapse Images.

Non-rigid registration of cell nuclei in time-lapse microscopy images can be achieved through estimating the deformation fields using optical flow methods. In contrast to local optical flow models employed in the existing non-rigid registration methods, we introduce approaches based on a global optical flow model. Our registration model consists of a data fidelity term and a regularization term. We compared different regularizers for the deformation fields and found that a convex quadratic function is more suitable than non-convex ones. To improve the robustness, we propose an adaptive weighting scheme based on the statistics of the noise in fluorescence microscopy images as well as a combined local-global scheme. Moreover, we extend the global method by exploiting high-order image features. The best suitable high-order features are determined through learning two generative image models, namely, fields of experts and convolutional Gaussian restricted Boltzmann machine, whose model formulations are both consistent with the assumption of high-order feature constancy in the registration model. Using multiple data sets of real 2D and 3D live cell microscopy image sequences as well as synthetic image data, we demonstrate that our proposed approach outperforms the previous methods in terms of both registration accuracy and computational efficiency.

Application of optical flow methods to aerated skimming flows above triangular and trapezoidal step cavities

ABSTRACTStepped chutes are built to provide safe flood passage in dams. The steps are associated with strong turbulence generation, which allow air to be entrained from the free surface. The present work adopts a non-intrusive local optical flow technique to examine the effects of partial cavity blockage on the mean and turbulent properties in aerated skimming flows in two large-size stepped chute models. The partial cavity blockage was found to be associated with decreased mean velocities in the overflow. The step edge was identified as the most significant source of turbulent production. General increases in turbulence levels and large scale turbulent motions were identified for the partially blocked cavities, which could be linked to a reduction in mutual sheltering between adjacent roughness elements. It was implied that the cavity blockage might have some effects on the velocity gradient fluctuations.