Vision-based Tracking System Research Articles

Vision-Based Target Tracking Controller Design for Asteroid Flyby Problem

An accurate target tracking system improves the quality of flyby imaging of the target and consequently aids effective planetary exploration. A vision-based tracking system is one of the more popular systems, utilizing imaging data to achieve the required tracking accuracy. However, recent high-accuracy target tracking requirements cannot be achieved with targeting error feedback only. This paper summarizes the asteroid flyby mission characteristics and requirements, shows how possible error sources in the tracking system, and displays an exemplar control system design. The tracking error source is generally divided into a navigation error, which is a tracking profile generation error, and a control error, which is caused by the control system. A high-quality onboard relative-trajectory determination system is required to improve the navigation accuracy of the control system, because the pre-designed or offline-estimated relative trajectory contains uncertainties that cause the tracking profile generation error. Previous studies focused on this navigation error because of its sensitive effect on control system performance, showing some methods for improvement. This paper discusses how the control system can be improved to achieve greater precision during target tracking by considering the navigation and system characteristics. A realistic case study scenario is defined and analyzed based on an actual target tracking interplanetary probe mission. The characteristics of the designed control system are presented and its performance is analyzed via numerical simulation, using the case study data.

Motion and viewing analysis during minimally invasive surgery for autostereoscopic visualization.

Autostereoscopic 3D visualization (ASV) forms a potentially appealing alternative to stereoscopic 3D displays to help surgeons regain depth perception during minimally invasive surgery (MIS). However, the feasibility of using single-viewer ASV has not yet been demonstrated in a clinical context. The purpose of the study is to analyze the current surgical workflow and display usage and assess the potential for using ASV in MIS applications. Additionally, the study seeks to acquire a better understanding of key design requirements, such as the eye-tracking performance and the lenticular lens 3D workspace. Two types of gynecologic interventions were investigated. A vision-based tracking system was developed, consisting of depth cameras mounted on the displays and ArUco markers placed on the hair caps of clinicians and the wall of the operating room. This allowed simultaneous tracking of the pose of operating staff and displays. Overall 20 surgeries were recorded, where 4 clinicians operated using 3 displays. Users were typically standing at a mean distance of 1900mm in a range from to 1200 to 2300mm from the display. Left-right motion was from -600 to 658mm. Clinicians stood on average 1000mm from each other. The head roll angle was below 16[Formula: see text]. Surgeons were looking predominantly (99%) to the same display. Observations took place from fairly well-defined places and with sufficient potential to differentiate between clinicians, suggesting that single-viewer ASV would be feasible.

Turning Movement Count Data Integration Methods for Intersection Analysis and Traffic Signal Design.

Traffic simulation is widely used for modeling, planning, and analyzing different strategies for traffic control and road development in a cost-efficient manner. In order to perform an intersection simulation, random vehicle trip data are typically applied to an intersection network, making them unrealistic. In this paper, we address this issue by presenting two different methods of incorporating actual turning movement count (TMC) data and comparing their similarity for intersection simulation and analysis. The TMC of three intersections in Las Vegas are estimated separately for one hour using a developed vision-based tracking system and they are incorporated into Simulation of Urban MObility (SUMO) for estimating traffic measurements and traffic signal design. t-tests with a 95% confidence interval on the simulation variables demonstrate the importance of using a route-based creation method which injects vehicles into a simulation environment based on the frame-level departure time. The intersection analyses and comparisons are performed based on estimated traffic measurements such as travel time, density, lane density, occupancy, and normalized waiting time. Since the critical edge of each intersection network is identified based on a higher normalized waiting time, new traffic signal designs are suggested based on the actual critical turning movements and improvements in vehicle travel time are achieved to better accommodate the actual traffic demand.

Siamese Transformer Network: Building an autonomous real-time target tracking system for UAV

Currently, the common used vision-based tracking system faces two major challenges: (1) the trade-off between speed and accuracy of the tracker; (2) the robustness of the tracking servo control. In this paper, we propose a new framework composed of a transformer attached to the siamese-type feature extraction networks called Siamese Transformer Network (SiamTrans) to balance the speed and accuracy, avoiding complicated hand-designed components and tedious post-processing of most existing siamese-type trackers with pre-defined anchor boxes or anchor-free schemes. SiamTrans forces the final set of predictions via bipartite matching, significantly reducing hyper-parameters associated with the candidate boxes. Moreover, to enhance the robustness of the servo control, the high-level control part is also redesigned by fusing all the bounding box information and with the Tracking Drift Suppression Strategy (TDSS). The TDSS is mainly used to judge the target’s loss. If the target is lost, it will feedback the previous information to reinitialize the tracker to track and update the template patch of SiamTrans, making the whole system more robust. Extensive experiments on visual tracking benchmarks, including GOT-10K, UAV123, demonstrate that SiamTrans achieves competitive performance and runs at 50 FPS. Specifically, SiamTrans outperforms the leading anchor-based tracker SiamRPN++ in the GOT-10K benchmark, confirming its effectiveness and efficiency. Furthermore, SiamTrans is deployed on the embedded device in which the algorithm can be run at 30FPS or 54FPS with TensorRT meeting the real-time requirements. In addition, we design the complete tracking system demo that can accurately track the target for multiple categories. The actual experimental results also show that the whole system is efficient and robust. The demo video link is as follows: https://youtu.be/UK37Q-M9ET4.

Real-time molecular imaging of near-surface tissue using Raman spectroscopy

The steady progress in medical diagnosis and treatment of diseases largely hinges on the steady development and improvement of modern imaging modalities. Raman spectroscopy has attracted increasing attention for clinical applications as it is label-free, non-invasive, and delivers molecular fingerprinting information of a sample. In combination with fiber optic probes, it also allows easy access to different body parts of a patient. However, image acquisition with fiber optic probes is currently not possible. Here, we introduce a fiber optic probe-based Raman imaging system for the real-time molecular virtual reality data visualization of chemical boundaries on a computer screen and the physical world. The approach is developed around a computer vision-based positional tracking system in conjunction with photometric stereo and augmented and mixed chemical reality, enabling molecular imaging and direct visualization of molecular boundaries of three-dimensional surfaces. The proposed approach achieves a spatial resolution of 0.5 mm in the transverse plane and a topology resolution of 0.6 mm, with a spectral sampling frequency of 10 Hz, and can be used to image large tissue areas in a few minutes, making it highly suitable for clinical tissue-boundary demarcation. A variety of applications on biological samples, i.e., distribution of pharmaceutical compounds, brain-tumor phantom, and various types of sarcoma have been characterized, showing that the system enables rapid and intuitive assessment of molecular boundaries.

A multi-resolution deep feature framework for dynamic displacement measurement of bridges using vision-based tracking system

Structural displacement is an imperative indicator for safety evaluation and maintenance. To address the limitations of conventional displacement sensors, advanced non-contact vision-based trackers offer a promising alternative. Based on the reconstructed Efficient Convolution Operator (ECO), a more powerful multi-resolution deep feature framework is proposed to efficiently encode the informative representation. The fusion of the shallow convolutional and the deep convolutional features is discriminative while preserving spatial and structural information. Furthermore, the discrete feature map is transferred to the continuous spatial domain by introducing an interpolation operator to achieve accurate sub-pixel registration. A careful comparison of the results on the steel suspension bridge demonstrates the high accuracy of the multi-resolution deep feature tracker (MDFT) for displacement measurement. The performances in the time and frequency domain show decent agreement with the results acquired by the laser displacement sensor (LDS), which confirm the low-cost, target-free, high resolution, and non-contact measurement capacities.

Fuzzy-PI double-layer stability control of an online vision-based tracking system

In many manufacturing processes (e.g., welding, spraying, coating adhesive), the control for the velocity in the main direction heavily affects operation quality. In addition to traditional manual operations, the industrial robot with a tracking system is capable of accurate and stable velocity control. In this paper, an intelligent robot tracking system is designed for implementing an appropriate velocity control and improving the performance of an autonomous system with online structured light vision tracking. For this aim, an effective tracking algorithm is proposed based on position-based visual servoing (PBVS), and motion compensation is implemented according to both detected path and taught path. To improve the adaptability of the system, a Fuzzy-PI double-layer controller is developed, which adjusts the movement of the end effector in both cases of large and small deviation. Welding experiments demonstrate the effectiveness of the proposed vision tracking system.

Autonomous Tracking by an Adaptable Scaled KCF Algorithm

A multicopter is equipped by a passive tracking device to follow a specified target. However, if want to track a non-controlled target, the passive tracking device is failed. We propose a vision-based tracking system for multicopters, used computer vision method to track any target without additional tracking devices. In this study, propose scale candidate graphs and scale tables to improve KCF. There are also stable results when the scale changes. The proposed an adaptable scaled KCF algorithm, when the KCF tracking failed, a feature-based matching detector is used to re-detect the target. Several experiments on various scene based on the proposed approach were conducted and evaluated. Stable tracking results were obtain to show the feasibility of the proposed system.

Surgical navigation system for laparoscopic lateral pelvic lymph node dissection in rectal cancer surgery using laparoscopic-vision-tracked ultrasonic imaging.

Laparoscopic lateral pelvic lymph node dissection (LPLND) in rectal cancer surgery requires considerable skill because the pelvic arteries, which need to be located to guide the dissection, are covered by other tissues and cannot be observed on laparoscopic views. Therefore, surgeons need to localize the pelvic arteries accurately before dissection, to prevent injury to these arteries. This report proposes a surgical navigation system to facilitate artery localization in laparoscopic LPLND by combining ultrasonic imaging and laparoscopy. Specifically, free-hand laparoscopic ultrasound (LUS) is employed to capture the arteries intraoperatively in this approach, and a laparoscopic vision-based tracking system is utilized to track the LUS probe. To extract the artery contours from the two-dimensional ultrasound image sequences efficiently, an artery extraction framework based on local phase-based snakes was developed. After reconstructing the three-dimensional intraoperative artery model from ultrasound images, a high-resolution artery model segmented from preoperative computed tomography (CT) images was rigidly registered to the intraoperative artery model and overlaid onto the laparoscopic view to guide laparoscopic LPLND. Experiments were conducted to evaluate the performance of the vision-based tracking system, and the average reconstruction error of the proposed tracking system was found to be 2.4mm. Then, the proposed navigation system was quantitatively evaluated on an artery phantom. The reconstruction time and average navigation error were 8min and 2.3mm, respectively. A navigation system was also successfully constructed to localize the pelvic arteries in laparoscopic and open surgeries of a swine. This demonstrated the feasibility of the proposed system in vivo. The construction times in the laparoscopic and open surgeries were 14 and 12min, respectively. The experimental results showed that the proposed navigation system can guide laparoscopic LPLND and requires a significantly shorter setting time than the state-of-the-art navigation systems do.

Occlusion-Based Coordination Protocol Design for Autonomous Robotic Shepherding Tasks

The robotic shepherding problem has earned significant research interest over the last few decades due to its potential application in precision agriculture. In this article, we first modeled the sheep flocking behavior using adaptive protocols and artificial potential field methods. Then, we designed a coordination algorithm for robotic dogs. An occlusion-based motion control strategy was proposed to herd the sheep to the desired location. Compared to formation-based techniques, the proposed control strategy provides more flexibility and efficiency when herding a large number of sheep. Simulation and lab-based experiments, using real robots and a global vision-based tracking system, were carried out to validate the effectiveness of the proposed approach.

Using Synthetic Data to Improve and Evaluate the Tracking Performance of Construction Workers on Site

Vision-based tracking systems enable the optimization of the productivity and safety management on construction sites by monitoring the workers’ movements. However, training and evaluation of such a system requires a vast amount of data. Sufficient datasets rarely exist for this purpose. We investigate the use of synthetic data to overcome this issue. Using 3D computer graphics software, we model virtual construction site scenarios. These are rendered for the use as a synthetic dataset which augments a self-recorded real world dataset. Our approach is verified by means of a tracking system. For this, we train a YOLOv3 detector identifying pedestrian workers. Kalman filtering is applied to the detections to track them over consecutive video frames. First, the detector’s performance is examined when using synthetic data of various environmental conditions for training. Second, we compare the evaluation results of our tracking system on real world and synthetic scenarios. With an increase of about 7.5 percentage points in mean average precision, our findings show that a synthetic extension is beneficial for otherwise small datasets. The similarity of synthetic and real world results allow for the conclusion that 3D scenes are an alternative to evaluate vision-based tracking systems on hazardous scenes without exposing workers to risks.

Real-time Safety Monitoring Vision System for Linemen in Buckets Using Spatio-temporal Inference

Linemen risk falls, electric shocks, burns, and other injuries during the daily job and these incidents can often be fatal. In this paper, we present a novel vision-based real-time system for detection and tracking of various non-rigid safety wearables worn by linemen, in a highly cluttered environment. We set up four imaging sensors on the repair truck’s bucket to robustly monitor the linemen from four different viewpoints. In the monitoring system, we firstly apply a novel fast background segmentation method to suppress false positives and reduce search space. Next, we represent each safety wearable with a Gaussian mixture model and track them with an LK-tracker. In order to track occluded or out-of-camera-view safety wearables, we propose a novel human pose inference method. The proposed method is an extension from the existing CNN-based human pose inference by utilizing light-weight color, shape, and space-based human pose inference mechanism. The proposed human pose inference method shows improved performance in terms of precision, recall, and speed. Experimental results on a number of challenging sequences demonstrate the effectiveness of the proposed scheme, under complex background, prolonged occlusions, and varying color, shape, and lighting.

Vision-based tracking system for augmented reality to localize recurrent laryngeal nerve during robotic thyroid surgery

We adopted a vision-based tracking system for augmented reality (AR), and evaluated whether it helped surgeons to localize the recurrent laryngeal nerve (RLN) during robotic thyroid surgery. We constructed an AR image of the trachea, common carotid artery, and RLN using CT images. During surgery, an AR image of the trachea and common carotid artery were overlaid on the physical structures after they were exposed. The vision-based tracking system was activated so that the AR image of the RLN followed the camera movement. After identifying the RLN, the distance between the AR image of the RLN and the actual RLN was measured. Eleven RLNs (9 right, 4 left) were tested. The mean distance between the RLN AR image and the actual RLN was 1.9 ± 1.5 mm (range 0.5 to 3.7). RLN localization using AR and vision-based tracking system was successfully applied during robotic thyroidectomy. There were no cases of RLN palsy. This technique may allow surgeons to identify hidden anatomical structures during robotic surgery.

A robust line-tracking photogrammetry method for uplift measurements of railway catenary systems in noisy backgrounds

For electrical railways, contact wires are used to ensure stable and continuous power supply to trains operating on the infrastructure. One crucial design parameter used to control the catenary design is the vertical displacement. The traditional uplift measuring equipment of catenary systems usually requires that targets are installed on the wire, and/or that measurement devices are mounted on the support structure of catenary systems or temporary poles. These methods require track access and substantial time and manpower for installation and measurement. Thus, a portable vision-based tracking system is proposed to enable remote, non-contact and non-target uplift measurements of catenary systems. The challenge in measuring the catenary displacement is to detect a linear rigid body in a noisy background environment with a high sampling frequency. To detect the displacement, a vision-based system with a novel line-tracking technique is used to achieve robust and accurate measurement. The proposed technique is based on a coarse subset and line search and subpixel detection. The line-tracking method is successfully employed for the identification of the important vertical displacements of a contact wire in front of a noisy background. The accuracy, robustness and applicability of the proposed tracking method are demonstrated through numerical experiments, the field uplift measurement of the railway catenary system and the identification of the contact wire in a diverse city environment.

Direction and distance sensors and sensing system for elderly people

Elderly people need either a caretaker or a cart to carry their belongings in and around their house or to/from nearby places, like a shopping mall, bus stop etc. This paper reports the development of a direction-cum-distance sensing system (DDSS) using ultrasonic sensors for a cart to assist elderly people. The DDSS can convert an ordinary assistive cart into a person-following assistive robot cart that would follow its user/target. Instead of using infrared or vision-based tracking systems for sensing direction and distance, as reported by other researchers, the DDSS reported here uses ultrasonic beam/pulses. The DDSS integrates the two tasks of finding the direction and measuring the distance in a single unit. For finding the direction of the target (user), an ultrasonic transmitter is placed on the back of the target, whereas two ultrasonic sensors are placed on the front of the cart. The distance of the target from the cart is measured using the classical technique of measuring the time taken by ultrasonic pulses to travel from the cart up to the target and back from the target to the cart after its reflection from the target. Performance of the DDSS in direction mode has been evaluated experimentally, which is very gratifying. Since the DDSS works with ultrasonic beam/pulses, the assistive robot cart can work not only indoor but also outdoor (where infrared emissions from the sun and other sources may cause mal-operation of infrared-based tracking systems) and in low-visibility areas (where camera-based tracking systems may fail to work). The new DDSS has additional advantages of simplicity, ready availability of components and low cost. Some suggestions for further work are included in the last section of the paper.

Resolving Colliding Larvae by Fitting ASM to Random Walker-Based Pre-Segmentations.

Drosophila melanogaster is an important model organism for research in neuro- and behavioral biology. Automated studies of their locomotion are crucial to link sensory input and neural processing to motor output which has led to numerous vision-based tracking systems. However, most of these approaches share the inability to segment the contours of colliding animals causing identity losses, appearing and disappearing animals, and the absence of posture and motion related measurements during the time of the collision. We present a novel collision resolution algorithm enabling an accurate contour segmentation of multiple touching Drosophila larvae. Our algorithm utilizes an adapted active shape model (ASM) to learn a low dimensional posture space which is fitted to random-walker generated pre-segmentations. We evaluate our collision resolution algorithm using three publicly available datasets and compare it with the current state-of-the-art methods. In addition, we introduce a refined dataset enabling a segmentation evaluation on the level of pixel accuracy. The results demonstrate that our approach outperforms the state-of-the-art approaches in both accuracy and computational time. We will incorporate this algorithm into our widely used tracking program to improve the statistical strength of the behavioral quantification and allow marker-free studies of interacting Drosophila larvae.

Trajectory prediction of vehicles turning at intersections using deep neural networks

In this paper, an early prediction of vehicle trajectories and turning movements are investigated using traffic cameras. A vision-based tracking system is developed to monitor intersection videos and collect vehicle trajectories with their labels known as turning movements. Firstly, two intersection videos are monitored for 2 h, and collected trajectories with their labels are used to train deep neural networks and obtain the turning models for the prediction task. Deep neural networks are further investigated on a third intersection with different video settings. The future 2 s evaluation of trajectories shows the success of long short-term memory networks to early predict the turning movements with more than 92% accuracy.

Efficient Pose Tracking based on Line Segment Matching

Pose tracking is a crucial issue for many applications such as robotic tasks and facility operations. Vision-based approaches with non-contact properties are appropriate choices for these tasks. However, vision-based approaches are not sufficiently robust and fast. In this work, we propose a vision-based pose tracking to deal with these problems. We estimate poses using Lie group and Lie algebra representation theory. Such operation is performed in a linearized space, therefore it is convenient for pose estimation. To provide reliable visual information for our pose estimation, we detect line segments. Our detection of line segment depends on semi-global image information. We describe all line segments and match those detected in consecutive frames. Our line segment detector and matching descriptor are good at discarding ambiguous line segments and finding real ones in noisy situations. The integration of group theory and line segment detection and matching plays an important role for developing a robust vision-based pose tracking system. Our system proves to be efficient and robust.

High-Throughput Analysis of Behavior Under the Control of Optogenetics in Caenorhabditis elegans.

In this unit, we describe an inexpensive and versatile method for optogenetic stimulation of a large population of genetically engineered Caenorhabditis elegans worms while quantitatively analyzing behavior. A custom light-emitting diode light source is used to deliver blue-light stimuli, causing direct depolarization of neurons expressing the light-gated cation channel Channelrhodopsin-2, which in turn evokes behavioral responses. The behavioral responses are recorded by a high-throughput machine vision-based tracking system, the Multi-Worm Tracker, for detailed analysis. This approach allows researchers to bypass technical obstacles to simultaneously deliver uniform stimuli to a large number of freely behaving animals and investigate the neural underpinnings of behavior. © 2018 by John Wiley & Sons, Inc.

GESTO: A Glove for Enhanced Sensing and Touching Based on Inertial and Magnetic Sensors for Hand Tracking and Cutaneous Feedback

The human hand represents a complex fascinating system with highly sensitive sensory capabilities and dexterous grasping and manipulation functionalities. As a consequence, estimating the hand pose and at the same time having the capability to provide haptic feedback in a wearable way may benefit areas such as rehabilitation, human–robot interaction, gaming, and many more. Existing solutions allow us to accurately measure the hand configuration and provide effective force feedback to the user. However, they have limited wearability/portability. In this paper, we present the wearable sensing/actuation system glove for enhanced sensing and touching (GESTO). It is based on inertial and magnetic sensors for hand tracking, coupled with cutaneous devices for the force feedback rendering. Unlike vision-based tracking systems, the sensing glove does not suffer from occlusion problems and lighting conditions. We properly designed the cutaneous devices in order to reduce possible interferences with the magnetic sensors and performed an experimental validation on ten healthy subjects. In order to measure the estimation accuracy of GESTO, we used a high-precision optical tracker. A comparison between using the glove with and without the haptic devices shows that the presence of them does not induce a statistically significant increase in the estimation error. Experimental results revealed the effectiveness of the proposed approach. The accuracy of our system, $3.32^\circ$ mean estimation error in the worst case, is comparable with the human ability of discriminating finger joint angle.