Real-time Multi-target Tracking Research Articles

A hybrid approach to real-time multi-target tracking

A hybrid approach to real-time multi-target tracking

Deep learning and multi-modal fusion for real-time multi-object tracking: Algorithms, challenges, datasets, and comparative study

Real-time multi-object tracking (MOT) is a complex task involving detecting and tracking multiple objects. After the objects are detected, they are assigned markers, and their trajectories are tracked in real-time. The scientific community is intrigued by the possibilities of utilizing MOT technology in the context of smart cities. Their primary focus lies in the domains of intelligent transportation, detection of vehicles and pedestrians, crowd surveillance, and public safety. Deep learning techniques have been developed in recent years to effectively tackle the challenges of real-time MOT tasks and enhance tracking performance. Environmental perception within smart traffic applications relies heavily on sensor data fusion. In traffic scenarios, a thoughtful approach involves utilizing a combination of sensors and cameras to detect and track targets while gathering valuable data effectively. However, it faces challenges when it comes to detecting and tracking objects that are in motion, have complex changes in appearance, or are in crowded scenes. This paper explores the foundational standard for real-time Multiple Object Tracking tasks. We prioritize the examination of quantitative measures by conducting a comprehensive analysis of widely utilized benchmark datasets and metrics. This study also investigates established embedding techniques and multi-modal fusion methods within real-time multi-target tracking algorithms. Each strategy will be classified and assessed according to a predefined set of principles. The paper presents a comprehensive analysis and visual representation of various MOT strategies. Finally, this paper aims to present an overview of the current challenges faced by the MOT mission, as well as the potential objectives that lie ahead.

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

Real-Time Multitarget Tracking for Panoramic Video Based on Dual Neural Networks for Multisensor Information Fusion

A multitarget real-time tracking system for panoramic video with multisensor information fusion dual neural networks is studied and implemented by combining dual neural networks, fused geometric features, and deep learning-based target real-time tracking algorithms. The motion model of multisensor information fusion is analyzed, the dual neural network perturbation model is introduced, and the state variables of the system are determined. Combined with the data structure model of multisensor information fusion, a least-squares-based optimization function is constructed. In this optimization function, the image sensor is constructed as an observationally constrained residual term based on the pixel plane, the inertial measurement unit is constructed as an observationally constrained residual term based on the motion model, and the motion relationship between sensors is constructed as a motion-constrained residual term to derive a mathematical model for real-time multitarget tracking of panoramic video. The simulation experimental results show that the integrated position estimation accuracy of the multisensor information fusion dual neural network-based panoramic video multitarget real-time tracking fusion algorithm is 10.5% higher than that of the optimal distributed panoramic video multitarget fusion algorithm without feedback. The results of the simulation experiments show that the integrated position estimation accuracy of the multitarget real-time tracking fusion algorithm for panoramic video improves by 10.96%, and the integrated speed estimation accuracy improves by 6.32% compared with the optimal distributed multitarget fusion algorithm without feedback. The proposed panoramic video multitarget real-time tracking algorithm based on the dual neural network can effectively improve the target tracking accuracy of the model on degraded frames (motion blur, target occlusion, out-of-focus, etc.), and the stability of the algorithm for target location and category detection is effectively improved by multiframe feature fusion. The research in this paper provides better technical support and research theory for panoramic video multitarget real-time tracking.

First experimental evaluation of multi-target multileaf collimator tracking during volumetric modulated arc therapy for locally advanced prostate cancer

PurposeLocally advanced and oligometastatic cancer patients require radiotherapy treatment to multiple independently moving targets. There is no existing commercial solution that can simultaneously track and treat multiple targets. This study experimentally implemented and evaluated a real-time multi-target tracking system for locally advanced prostate cancer. MethodsReal-time multi-target MLC tracking was integrated with 3D x-ray image guidance on a standard linac. Three locally advanced prostate cancer treatment plans were delivered to a static lymph node phantom and dynamic prostate phantom that reproduced three prostate trajectories. Treatments were delivered using multi-target MLC tracking, single-target MLC tracking, and no tracking. Doses were measured using Gafchromic film placed in the dynamic and static phantoms. Dosimetric error was quantified by the 2%/2 mm gamma failure rate. Geometric error was evaluated as the misalignment between target and aperture positions. The multi-target tracking system latency was measured. ResultsThe mean (range) gamma failure rates for the prostate and lymph nodes, were 18.6% (5.2%, 28.5%) and 7.5% (1.1%, 13.7%) with multi-target tracking, 7.9% (0.7%, 15.4%) and 37.8% (18.0%, 57.9%) with single-target tracking, and 38.1% (0.6%, 75.3%) and 37.2% (29%, 45.3%) without tracking. Multi-target tracking had the lowest geometric error with means and standard deviations within 0.2 ± 1.5 for the prostate and 0.0 ± 0.3 mm for the lymph nodes. The latency was 730 ± 20 ms. ConclusionThis study presented the first experimental implementation of multi-target tracking to independently track prostate and lymph node displacement during VMAT. Multi-target tracking reduced dosimetric and geometric errors compared to single-target tracking and no tracking.

A Real-time Multi-target tracking method based on Deep Learning

In view of complex model and poor real-time performance of current multi-target tracking algorithms, a real-time online multi-target tracking method based on deep learning is proposed. Firstly, the detector is used to detect the target in video image space and obtain its detection frame. After that, the position, coordinates and motion of the next frame of the target are predicted by the Kalman filter. Then Complete Intersection over Union (CIOU) is used as the distance measure to calculate the overlap between the detection box and the prediction box, and further generate the similarity matrix. Finally, the Hungarian algorithm finds optimal matching in the bipartite graph formed by the two boxes, so as to realize the data association between multiple targets. Experimental evaluation shows that this method finishes real-time and online multi-target tracking well.

Strong Tracking PHD Filter Based on Variational Bayesian with Inaccurate Process and Measurement Noise Covariance.

Assuming that the measurement and process noise covariances are known, the probability hypothesis density (PHD) filter is effective in real-time multi-target tracking; however, noise covariance is often unknown and time-varying for an actual scene. To solve this problem, a strong tracking PHD filter based on Variational Bayes (VB) approximation is proposed in this paper. The measurement noise covariance is described in the linear system by the inverse Wishart (IW) distribution. Then, the fading factor in the strong tracking principle uses the optimal measurement noise covariance at the previous moment to control the state prediction covariance in real-time. The Gaussian IW (GIW) joint distribution adopts the VB approximation to jointly return the measurement noise covariance and the target state covariance. The simulation results show that, compared with the traditional Gaussian mixture PHD (GM-PHD) and the VB-adaptive PHD, the proposed algorithm has higher tracking accuracy and stronger robustness in a more reasonable calculation time.

A Local Environment Model Based on Multi-Sensor Perception for Intelligent Vehicles

Accurate perception of the driving environment is a key technology for intelligent vehicles. Given some critical problems such as low robustness, low detection precision, difficulty in actual deployment, we propose a local environment model (LEM) based on multi-sensor fusion technology through Lidar, millimeter-wave (MMW) radar, camera, and ultrasonic radar. The local environment model mainly consists of the drivable area and the dynamic target list. The drivable area is extracted by the ground gradient threshold algorithm. Based on it, we propose an effective trim algorithm to make the drivable area model more practical. Furthermore, low-cost ultrasonic radars are deployed to compensate for the blind area of Lidar. The dynamic target list is established by local tracking and global tracking in the forward area. Kalman filter and converted measurement Kalman filter (CMKF) are adopted in the local tracking of Lidar, camera, and MMW radar. In the global tracking, the global nearest neighbor (GNN) algorithm is used for data association and the optimal distributed estimation fusion (ODEF) algorithm is used for sensor fusion. To improve the robustness of tracking, we use an assignment method to better exploit sensor performance. Finally, the vehicle experiment is carried out in the campus environment. Experimental results indicate that the proposed algorithm can avoid the false detection of the drivable area and realize real-time multi-target dynamic tracking. Therefore, the robustness and accuracy of the local environment model is verified.

Optimization of Particle CBMeMBer Filters for Hardware Implementation

It is a promising solution for real-time multitarget tracking to implement particle cardinality balanced multitarget multi-Bernoulli (CBMeMBer) filters in hardware platforms. However, this solution is difficult to materialize since there is a contradiction between the time-varying number of Bernoulli intensity components in CBMeMBer filters and the limited number of particles in hardware platforms. Moreover, real-time hardware implementation requires a resampling procedure that is suitable for parallel processing, while the existing parallel resampling algorithms oversimplify this procedure, resulting in estimation performance degradation. In this paper, we propose an optimization algorithm of particle allocation to overcome the above-mentioned contradiction, and a parallel resampling algorithm to improve the estimation performance. Numerical experiments demonstrate the effectiveness of the proposed algorithms in multitarget tracking.

Real-time multitarget tracking for sensor-based sorting

Utilizing parallel algorithms is an established way of increasing performance in systems that are bound to real-time restrictions. Sensor-based sorting is a machine vision application for which firm real-time requirements need to be respected in order to reliably remove potentially harmful entities from a material feed. Recently, employing a predictive tracking approach using multitarget tracking in order to decrease the error in the physical separation in optical sorting has been proposed. For implementations that use hard associations between measurements and tracks, a linear assignment problem has to be solved for each frame recorded by a camera. The auction algorithm can be utilized for this purpose, which also has the advantage of being well suited for parallel architectures. In this paper, an improved implementation of this algorithm for a graphics processing unit (GPU) is presented. The resulting algorithm is implemented in both an OpenCL and a CUDA based environment. By using an optimized data structure, the presented algorithm outperforms recently proposed implementations in terms of speed while retaining the quality of output of the algorithm. Furthermore, memory requirements are significantly decreased, which is important for embedded systems. Experimental results are provided for two different GPUs and six datasets. It is shown that the proposed approach is of particular interest for applications dealing with comparatively large problem sizes.

Influence of robotic shoal size, configuration, and activity on zebrafish behavior in a free-swimming environment

In animal studies, robots have been recently used as a valid tool for testing a wide spectrum of hypotheses. These robots often exploit visual or auditory cues to modulate animal behavior. The propensity of zebrafish, a model organism in biological studies, toward fish with similar color patterns and shape has been leveraged to design biologically inspired robots that successfully attract zebrafish in preference tests. With an aim of extending the application of such robots to field studies, here, we investigate the response of zebrafish to multiple robotic fish swimming at different speeds and in varying arrangements. A soft real-time multi-target tracking and control system remotely steers the robots in circular trajectories during the experimental trials. Our findings indicate a complex behavioral response of zebrafish to biologically inspired robots. More robots produce a significant change in salient measures of stress, with a fast robot swimming alone causing more freezing and erratic activity than two robots swimming slowly together. In addition, fish spend more time in the proximity of a robot when they swim far apart than when the robots swim close to each other. Increase in the number of robots also significantly alters the degree of alignment of fish motion with a robot. Results from this study are expected to advance our understanding of robot perception by live animals and aid in hypothesis-driven studies in unconstrained free-swimming environments.

An Efficient Data-Driven Particle PHD Filter for Multitarget Tracking

In this paper, we propose an efficient data-driven particle probability hypothesis density (PHD) filter for real-time multitarget tracking of nonlinear/non-Gaussian system in dense clutter environment. In specific, the input measurements are first classified into two sets, namely survival measurements and spontaneous birth measurements, after eliminating clutters by using existing historic state data of targets. Since most clutters do not participate in the complex weight computation of particle PHD filter, better real-time performance can be achieved. The tracking performance is also improved because the survival measurements are used for survival targets and the spontaneous birth measurements are used for spontaneous birth targets, resulting in less interference from each other and from clutters. Extensive simulations validate the improvement of both the real-time performance and tracking performance of the proposed data-driven particle PHD filter in comparison with the traditional particle PHD filter.

Research on Real-time Multi-target Tracking Algorithm Based on MSPF

Recently,real-time monitoring multi-target tracking as an important component of intelligent transportation system(ITS) has been paid much attention.The traditional multitarget tracking algorithm has problems that the processing speed is slow and the false matches may happen when vehicles cross.Firstly,the algorithm detects moving targets through modeling a complex background based on Bayesian rules,then introduces a multi-target tracking algorithm based on mean shift particle filter(MSPF).Firstly,the algorithm predicts the extent possible by the use of MSPF for each vehicle in the next frame,uses different detection strategies for simple or multiple targets to avoid a global search and improve the tracking speed;by constructing the importance density function based on the latest observations,the algorithm can achieve an accurate and robust tracking in the part of the block and cross-vehicle.Simulation results verify the proposed algorithm.

A Scalable Multi-Target Tracking Algorithm for Wireless Sensor Networks

Multi-target tracking is a representative real-time application of sensor networks as it exhibits different aspects of sensor networks such as event detection, sensor information fusion, multihop communication, sensor management, and real-time decision making. The task of tracking multiple objects in a wireless sensor network is challenging due to constraints on a sensor node such as short communication and sensing ranges, a limited amount of memory, and limited computational power. In addition, since a sensor network surveillance system needs to operate autonomously without human operators, it requires an autonomous real-time tracking algorithm which can track an unknown number of targets. In this paper, we develop a scalable real-time multi-target tracking algorithm that is autonomous and robust against transmission failures, communication delays, and sensor localization error. The algorithm is based on a rigorous probabilistic model and an approximation scheme for the optimal Bayesian filter. In particular, an extensive simulation study shows that there is no performance loss up to an average localization error of 0.7 times the separation between sensors and the algorithm tolerates up to 50% lost-to-total packet ratio and 90% delayed-to-total packet ratio. The proposed algorithm has been successfully applied to real-time multi-target tracking problems using wireless sensor networks.

Improved Method Based on MPF for Multi-target Tracking

Particle filter algorithm is poor at consistently maintaining the multi-modality problem and remains particle degeneracy phenomenon. This shortcoming can be addressed through using Mixture Particle Filter (MPF) to model the target distribution as a non-parametric mixture model. According to computational complexity of the particle filter algorithm, Gaussian Particle Filter (GPF) is used. The particle set is obtained by sampling of Gaussian density function instead of re-sampling to improve the computing speed of particle filter algorithm. In this paper, combined with the advantages of GPF and MPF, GM-MPF algorithm is proposed to improve the real-time of MPF. The experimental results show that GM-MPF algorithm can effectively solve the problem of multi-modality, particle degradation and achieve real-time multi-target tracking.

A real-time multitarget tracking system with robust multichannel CNN-UM algorithms

This paper introduces a tightly coupled topographic sensor-processor and digital signal processor (DSP) architecture for real-time visual multitarget tracking (MTT) applications. We define real-time visual MTT as the task of tracking targets contained in an input image flow at a sampling-rate that is higher than the speed of the fastest maneuvers that the targets make. We utilize a sensor-processor based on the cellular neural network universal machine architecture that permits the offloading of the main image processing tasks from the DSP and introduces opportunities for sensor adaptation based on the tracking performance feedback from the DSP. To achieve robustness, the image processing algorithms running on the sensor borrow ideas from biological systems: the input is processed in different parallel channels (spatial, spatio-temporal and temporal) and the interaction of these channels generates the measurements for the digital tracking algorithms. These algorithms (running on the DSP) are responsible for distance calculation, state estimation, data association and track maintenance. The performance of the proposed system is studied using actual hardware for different video flows containing rapidly moving maneuvering targets.

Real-time multitarget tracking by a cooperative distributed vision system

Target detection and tracking is one of the most important and fundamental technologies to develop real-world computer vision systems such as security and traffic monitoring systems. This paper first categorizes target tracking systems based on characteristics of scenes, tasks, and system architectures. Then we present a real-time cooperative multitarget tracking system. The system consists of a group of active vision agents (AVAs), where an AVA is a logical model of a network-connected computer with an active camera. All AVAs cooperatively track their target objects by dynamically exchanging object information with each other With this cooperative tracking capability, the system as a whole can track multiple moving objects persistently even under complicated dynamic environments in the real world. In this paper we address the technologies employed in the system and demonstrate their effectiveness.

Optimizing Murty's ranked assignment method

We describe an implementation of an algorithm due to Murty for determining a ranked set of solutions to assignment problems. The intended use of the algorithm is in the context of multitarget tracking, where it has been shown that real-time multitarget tracking is feasible for some problems, but many other uses of the algorithm are also possible. The following three optimizations are discussed: (1) inheriting dual variables and partial solutions during partitioning, (2) sorting subproblems by lower cost bounds before solving, and (3) partitioning in an optimized order. When used to find the 100 best solutions to random 100/spl times/100 assignment problems, these optimizations produce a speedup of over a factor of 20, finding all 100 solutions in about 0.6 s. For a random cost matrix, the average time complexity for finding k solutions to random N/spl times/N problems appears to be nearly linear in both k and N, for sufficiently large k.