Lucas-Kanade Tracker Research Articles

River video flow measurement algorithm with space-time image fusion of object tracking and statistical characteristics

River flow monitoring serves as a crucial data source in hydrological stations, employing the widely adopted space-time image velocimetry (STIV) method both domestically and internationally. However, current methods yield space-time (ST) image trajectories with limited usable information, particularly due to the interference of complex flow characteristics and harsh outdoor environments in natural rivers. This interference leads to errors in detecting the main orientation of texture, consequently reducing measurement accuracy. To overcome these challenges, this paper proposes an river video flow measurement algorithm with ST image fusion of object tracking and statistical characteristics. The Lucas–Kanade tracking algorithm is employed to track velocity points and generate tracking trajectories. These trajectories’ coordinates are then fitted into straight lines to automatically arrange velocity-measuring lines in natural rivers. The algorithm leverages multi-scale detail boosting and Gaussian directional stretch filtering to enhance the quality and texture trajectories of ST images. Furthermore, background noise filtering and stripe noise filtering are applied to denoise the ST images, effectively mitigating the impact of noise caused by the natural river environment and camera equipment. The proposed algorithm incorporates projection statistical characteristics algorithms to accurately detect the dominant texture direction, enabling precise calculation of mean velocity and cross-sectional discharge of the river. Experimental validation was conducted in both natural and regular river channels, demonstrating the superior performance of the proposed solution compared to widely used large-scale particle image velocimetry, STIV, MobileNet-STIV and FD-DIS-G algorithms. The algorithm exhibited higher accuracy, stability, and applicability, with errors at each velocity-measuring point below 10%, and relative errors of the calculated mean velocity and cross-sectional discharge below 3%. This affirms the algorithm’s superior accuracy, stability, and applicability.

Vision technique for deflection measurements based on laser positioning

This paper proposes a vision-based target-free deflection measurement method of bridge. This method eliminates the trouble of installing artificial targets on the structure. The proposed method utilizes multi-point laser projection as a reference point, which can eliminate the error caused by vibration at the camera position, and also avoids the difficulty of finding the background building as a reference point. The proposed method controls the measurement accuracy by adjusting the size of the captured video area. Through the method of camera scanning, large-scale multi-point deflection measurement is carried out under the premise of certain precision. The video laser spot reference point (LSRP) before loading is selected by the image segmentation algorithm. The Oriented features from accelerated cegment Test (FAST) and rotated binary robust independent elementary features (BRIEF) (ORB) feature detection and extraction algorithm is used to detect and extract key features in the region of interest (ROI) near the LSRP. The extracted key feature points calculate the displacement of each frame of the entire video by using the Lucas-Kanade Tracking (KLT) algorithm, which ensures the maximum number of matches of key points between consecutive images. The mismatched outliers are discarded by using the Random Sampling Consistency (RANSAC) algorithm. The accuracy of the proposed method was verified by an indoor and outdoor beam experiment. In the experiment, the results obtained by the proposed method are compared with the vertical displacement values of the corresponding positions acquired by the displacement meter obtained by the high-precision DH3816N dynamic strain test system to verify the validity and accuracy of the proposed method.

Registration of retinal sequences from new video-ophthalmoscopic camera.

BackgroundAnalysis of fast temporal changes on retinas has become an important part of diagnostic video-ophthalmology. It enables investigation of the hemodynamic processes in retinal tissue, e.g. blood-vessel diameter changes as a result of blood-pressure variation, spontaneous venous pulsation influenced by intracranial-intraocular pressure difference, blood-volume changes as a result of changes in light reflection from retinal tissue, and blood flow using laser speckle contrast imaging. For such applications, image registration of the recorded sequence must be performed.MethodsHere we use a new non-mydriatic video-ophthalmoscope for simple and fast acquisition of low SNR retinal sequences. We introduce a novel, two-step approach for fast image registration. The phase correlation in the first stage removes large eye movements. Lucas-Kanade tracking in the second stage removes small eye movements. We propose robust adaptive selection of the tracking points, which is the most important part of tracking-based approaches. We also describe a method for quantitative evaluation of the registration results, based on vascular tree intensity profiles.ResultsThe achieved registration error evaluated on 23 sequences (5840 frames) is 0.78 ± 0.67 pixels inside the optic disc and 1.39 ± 0.63 pixels outside the optic disc. We compared the results with the commonly used approaches based on Lucas-Kanade tracking and scale-invariant feature transform, which achieved worse results.ConclusionThe proposed method can efficiently correct particular frames of retinal sequences for shift and rotation. The registration results for each frame (shift in X and Y direction and eye rotation) can also be used for eye-movement evaluation during single-spot fixation tasks.

Salient Point Tracking for Key Frames Selection in Outdoor Image Sequences

Key frames are of great importance in applications where it is necessary to reduce information redundancy in the images, estimate a moving camera's pose efficiently, or make precise 3D reconstructions. Realtime key frame selection is commonly done by tracking many salient points until a certain percentage is no longer visible. The Fundamental-based tracker is an algorithm that maintains consistent correspondences through successive frames using the Fundamental matrix and the Epipolar geometry. We present an algorithm for tracking hundreds of salient points in sequences of monocular images of exterior urban and aquatic scenes. We extend the Fundamental- based tracker to discover points that are not extracted by the local feature detector because these points are below a threshold of detection. We employ the information of the scene geometry, the image appearance, and the apparent motion dynamics to find these points. This proposal allows maintaining many stable tracks for a longer time. We carry out an experimental evaluation on three very different image sequences captured by a single camera mounted on wheeled and underwater vehicles. Our algorithm shows a reliable tracking performance compared with the Pyramidal Lucas-Kanade tracker, at processing rates of 4 to 9 frames per second on a laptop.

Learning based particle filtering object tracking for visible-light systems

We propose a novel object tracking framework based on online learning scheme that can work robustly in challenging scenarios. Firstly, a learning-based particle filter is proposed with color and edge-based features. We train a support vector machine (SVM) classifier with object and background information and map the outputs into probabilities, then the weight of particles in a particle filter can be calculated by the probabilistic outputs to estimate the state of the object. Secondly, the tracking loop starts with Lucas–Kanade (LK) affine template matching and follows by learning-based particle filter tracking. Lucas–Kanade method estimates errors and updates object template in the positive samples dataset, and learning-based particle filter tracker will start if the LK tracker loses the object. Finally, SVM classifier evaluates every tracked appearance to update the training set or restart the tracking loop if necessary. Experimental results show that our method is robust to challenging light, scale and pose changing, and test on eButton image sequence also achieves satisfactory tracking performance.

Predictive Model Reference Adaptive Controller to Compensate Heart Motion in Minimally Invasive CABG Surgery.

Heart beating is a major challenge in minimally invasive coronary artery surgery. A promising solution is to develop a motion compensation robotic system that gives the surgeon an impression of operating on motionless tissue by synchronizing the surgical tool automatically with the heart tissue motion. To achieve higher control accuracy, an intelligent controller called Predictive Model Reference Adaptive Controller is presented herein, which is adapted not only by observed reference signals but, also by unknown reference signals that are not observed by a camera but could be predicted by a state space estimator. To develop such a system, first the heart surface motion is tracked by the Lucas-Kanade tracking algorithm and validated by human observation. The results of implementing this control algorithm on a real human heart data set show capability of achieving a motion compensation system with high control accuracy.

Robust feature tracking on the beating heart for a robotic-guided endoscope

Visualization during minimally invasive bypass surgery on the beating heart can be enhanced by using a robotic-guided endoscope and visual servoing from the endoscopic images. In order to achieve these objectives, this work has focused on developing and testing algorithms for accurate, robust and real-time motion tracking of features on the beating heart, using marker-less approaches and an uncalibrated endoscope. Lucas-Kanade pyramidal optical flow-based algorithms and speeded-up robust features (SURF)-based methods have been extensively evaluated, using a range of developed metrics, in order to quantify accuracy, robustness and drift under a variety of circumstances. Three sets of experiments are reported: the first set compared the two tracking methods, using a beating-heart phantom and a static endoscope; the second set evaluated the methods when images were taken using a moving robotic-guided endoscope; and finally, the Lucas-Kanade optical flow algorithm was extensively tested in a visual servoing application, using a robotic endoscope. The combination of a Lucas-Kanade tracking algorithm and a SURF-based feature detection method gave the best performance in terms of accuracy and robustness of tracking, while preserving real-time computation requirements. The optimal parameters consist of a window size of 51 × 51 pixels and an interframe motion threshold of 20 pixels. Feature tracking was successfully integrated into uncalibrated visual servoing or a robotic-guided endoscope. Robust feature tracking on a beating heart with endoscopic video can be achieved in real-time and may facilitate robotically-assisted, minimally invasive bypass surgery and conventional laparoscopic surgery.

Environmental Security in Arctic Ice-Covered Seas: From Strategy to Tactics of Hazard Identification and Emergency Response

AbstractEnvironmental change and increasing industrial activity in the maritime Arctic require strategies to adapt to change and ensure safe operations. This problem has been defined at the broader strategic level. We evaluate key aspects of environmental security in ice-covered waters, focusing on tactical and operational information needs, which have received less attention. Monitoring of environmental hazards and effective emergency response in sea ice environments require high-resolution data of ice hazard distributions (e.g., multiyear ice, landfast ice breakout, and ice push events), ice movement and deformation, as well as ice characteristics and dynamics relevant to emergency response. We have developed a prototype coastal observing system at Barrow, Alaska, that addresses such information needs. Imagery obtained from a marine X-band radar with a digital controller is combined with data from on-ice sensors (ice thickness, ice and water temperature, sea level) and assessments of potentially hazardous ice conditions by local experts. Digital imagery and data are processed and disseminated in near-real time. Using a combination of image processing approaches (optical flow, Lucas-Kanade tracker), ice velocity fields, floe trajectories, and boundaries of stationary ice are derived automatically. Early onset of hazardous events is detected through Hidden Markov Modeling, providing potential decision support in operational settings. We evaluate the utility of the system and strategies towards integration with broader emergency response efforts.

Extending 3D Lucas–Kanade tracking with adaptive templates for head pose estimation

The Lucas–Kanade tracker (LKT) is a commonly used method to track target objects over 2D images. The key principle behind the object tracking of an LKT is to warp the object appearance so as to minimize the difference between the warped object’s appearance and a pre-stored template. Accordingly, the 2D pose of the tracked object in terms of translation, rotation, and scaling can be recovered from the warping. To extend the LKT for 3D pose estimation, a model-based 3D LKT assumes a 3D geometric model for the target object in the 3D space and tries to infer the 3D object motion by minimizing the difference between the projected 2D image of the 3D object and the pre-stored 2D image template. In this paper, we propose an extended model-based 3D LKT for estimating 3D head poses by tracking human heads on video sequences. In contrast to the original model-based 3D LKT, which uses a template with each pixel represented by a single intensity value, the proposed model-based 3D LKT exploits an adaptive template with each template pixel modeled by a continuously updated Gaussian distribution during head tracking. This probabilistic template modeling improves the tracker’s ability to handle temporal fluctuation of pixels caused by continuous environmental changes such as varying illumination and dynamic backgrounds. Due to the new probabilistic template modeling, we reformulate the head pose estimation as a maximum likelihood estimation problem, rather than the original difference minimization procedure. Based on the new formulation, an algorithm to estimate the best head pose is derived. The experimental results show that the proposed extended model-based 3D LKT achieves higher accuracy and reliability than the conventional one does. Particularly, the proposed LKT is very effective in handling varying illumination, which cannot be well handled in the original LKT.

Tracking by an Optimal Sequence of Linear Predictors

We propose a learning approach to tracking explicitly minimizing the computational complexity of the tracking process subject to user-defined probability of failure (loss-of-lock) and precision. The tracker is formed by a Number of Sequences of Learned Linear Predictors (NoSLLiP). Robustness of NoSLLiP is achieved by modeling the object as a collection of local motion predictors--object motion is estimated by the outlier-tolerant RANSAC algorithm from local predictions. Efficiency of the NoSLLiP tracker stems from (i) the simplicity of the local predictors and (ii) from the fact that all design decisions--the number of local predictors used by the tracker, their computational complexity (i.e. the number of observations the prediction is based on), locations as well as the number of RANSAC iterations are all subject to the optimization (learning) process. All time-consuming operations are performed during the learning stage--tracking is reduced to only a few hundreds integer multiplications in each step. On PC with 1xK8 3200+, a predictor evaluation requires about 30 microseconds. The proposed approach is verified on publicly-available sequences with approximately 12000 frames with ground-truth. Experiments demonstrates, superiority in frame rates and robustness with respect to the SIFT detector, Lucas-Kanade tracker and other trackers.

Anytime learning for the NoSLLiP tracker

We propose an anytime learning procedure for the Sequence of Learned Linear Predictors (SLLiP) tracker. Since learning might be time-consuming for large problems, we present an anytime learning algorithm which, after a very short initialization period, provides a solution with defined precision. As SLLiP tracking requires only a fraction of the processing power of an ordinary PC, the learning can continue in a parallel background thread continuously delivering improved, i.e. faster, SLLiPs with lower computational complexity and the same precision. The proposed approach is verified on publicly-available sequences with approximately 12,000 ground-truthed frames. The learning time is shown to be 20 times smaller than standard SLLiP learning based on linear programming, yet its robustness and accuracy is similar. Superiority in the frame-rate and robustness in comparison with the SIFT detector, Lucas–Kanade tracker and Jurie’s tracker is also demonstrated.