Multi-object Tracking System Research Articles

Multi-Scene Generalized Trajectory Global Graph Solver with Composite Nodes for Multiple Object Tracking

The global multi-object tracking (MOT) system can consider interaction, occlusion, and other ``visual blur'' scenarios to ensure effective object tracking in long videos. Among them, graph-based tracking-by-detection paradigms achieve surprising performance. However, their fully-connected nature poses storage space requirements that challenge algorithm handling long videos. Currently, commonly used methods are still generated trajectories by building one-forward associations across frames. Such matches produced under the guidance of first-order similarity information may not be optimal from a longer-time perspective. Moreover, they often lack an end-to-end scheme for correcting mismatches. This paper proposes the Composite Node Message Passing Network (CoNo-Link), a multi-scene generalized framework for modeling ultra-long frames information for association. CoNo-Link's solution is a low-storage overhead method for building constrained connected graphs. In addition to the previous method of treating objects as nodes, the network innovatively treats object trajectories as nodes for information interaction, improving the graph neural network's feature representation capability. Specifically, we formulate the graph-building problem as a top-k selection task for some reliable objects or trajectories. Our model can learn better predictions on longer-time scales by adding composite nodes. As a result, our method outperforms the state-of-the-art in several commonly used datasets.

PSMOT: Online Occlusion-Aware Multi-Object Tracking Exploiting Position Sensitivity.

Models based on joint detection and re-identification (ReID), which significantly increase the efficiency of online multi-object tracking (MOT) systems, are an evolution from separate detection and ReID models in the tracking-by-detection (TBD) paradigm. It is observed that these joint models are typically one-stage, while the two-stage models become obsolete because of their slow speed and low efficiency. However, the two-stage models have naive advantages over the one-stage anchor-based and anchor-free models in handling feature misalignment and occlusion, which suggests that the two-stage models, via meticulous design, could be on par with the state-of-the-art one-stage models. Following this intuition, we propose a robust and efficient two-stage joint model based on R-FCN, whose backbone and neck are fully convolutional, and the RoI-wise process only involves simple calculations. In the first stage, an adaptive sparse anchoring scheme is utilized to produce adequate, high-quality proposals to improve efficiency. To boost both detection and ReID, two key elements-feature aggregation and feature disentanglement-are taken into account. To improve robustness against occlusion, the position-sensitivity is exploited, first to estimate occlusion and then to direct the post-process for anti-occlusion. Finally, we link the model to a hierarchical association algorithm to form a complete MOT system called PSMOT. Compared to other cutting-edge systems, PSMOT achieves competitive performance while maintaining time efficiency.

Adaptive multi-object tracking based on sensors fusion with confidence updating

Multi-object tracking (MOT) systems typically rely on object detection results for tracking, so the accuracy of the MOT system is significantly affected by the error of the detector. Changes in error usually lead to unstable tracking. Regarding this problem, we proposed an adaptive MOT method based on detection confidence. At first, we use a simple data fusion method to combine the detection results of LiDAR and camera to reduce the large number of false detections. And then we used a factor based on confidence to adjust the estimating covariance matrix and measurement covariance matrix adaptively. The algorithm can judge which is more reliable between prediction and detection, and choose which is more important in the update step. Meanwhile, we set a factor based on confidence to control the search range in the data association module. Our method reduces the impact of detector error while ensuring accuracy and speed, and improves the robustness of the MOT algorithm. Through experiments conducted on the KITTI multi-object tracking dataset, our method has demonstrated significant advantages over state-of-the-art (SOTA) methods in terms of both accuracy and processing speed. The results of MOTA for 90.02% and FPS for 262.

Jointing Recurrent Across-Channel and Spatial Attention for Multi-Object Tracking With Block-Erasing Data Augmentation

Although deep-learning-based multi-object tracking (MOT) approaches have achieved remarkable performances in terms of accuracy and efficiency, the issue of object occlusions remains an open challenge for most one-shot MOT methods. To address the problem of object occlusions, in this paper we present a recurrent across-channel and spatial attention-based one-shot multi-object tracking method with block-erasing data augmentation. First, we construct a multiattention feature learning module, named RASFL, that combines recurrent across-channel attention with spatial attention. The RASFL extracts both the correlations of the feature channels and the differences of the spatial locations to improve the accuracy of the re-identification (Re-ID) task. Second, we adopt a block- erasing data augmentation strategy to handle object occlusions by using random pixel blocks to simulate occlusion cases during the network training process. This block-erasing data augmentation assists the network to be more robust under object occlusions. By integrating the proposed RASFL module and the block-erasing data augmentation strategy into a one-shot online MOT system, we build an accurate and robust MOT model called DcMOT. Finally, we run our method on the MOT16, MOT17 and MOT20 datasets to conduct a comprehensive comparison with some of the state-of-the-art MOT methods. The experimental results demonstrate that the proposed DcMOT model achieves a competitive performance in terms of both accuracy and efficiency; with especially good performances in the occlusion cases.

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

Simulation study and experimental validation of a neural network-based predictive tracking system for sensor-based sorting

Abstract Sensor-based sorting offers cutting-edge solutions for separating granular materials. The line-scanning sensors currently in use in such systems only produce a single observation of each object and no data on its movement. According to recent studies, using an area-scan camera has the potential to reduce both characterization and separation error in a sorting process. A predictive tracking approach based on Kalman filters makes it possible to estimate the followed paths and parametrize a unique motion model for each object using a multiobject tracking system. While earlier studies concentrated on physically-motivated motion models, it has been demonstrated that novel machine learning techniques produce predictions that are more accurate. In this paper, we describe the creation of a predictive tracking system based on neural networks. The new algorithm is applied to an experimental sorting system and to a numerical model of the sorter. Although the new approach does not yet fully reach the achieved sorting quality of the existing approaches, it allows the use of the general method without requiring expert knowledge or a fundamental understanding of the parameterization of the particle motion model.

Transformer for multiple object tracking: Exploring locality to vision

Multi-object tracking (MOT) is a critical task in various domains, such as traffic analysis, surveillance, and autonomous vehicles. The joint-detection-and-tracking paradigm has been extensively researched, which is faster and more convenient for training and deploying over the classic tracking-by-detection paradigm while achieving state-of-the-art performance. This paper explores the possibilities of enhancing the MOT system by leveraging the prevailing convolutional neural network (CNN) and a novel vision transformer technique Locality. There are several deficiencies in the transformer adopted for computer vision tasks. While the transformers are good at modeling global information for a long embedding, the locality mechanism, which learns the local features, is missing. This could lead to negligence of small objects, which may cause security issues. We combine the TransTrack MOT system with the locality mechanism inspired by LocalViT and find that the locality-enhanced system outperforms the baseline TransTrack by 5.3% MOTA on the MOT17 dataset.

An improved deep learning architecture for multi-object tracking systems

Robust and reliable 3D multi-object tracking (MOT) is essential for autonomous driving in crowded urban road scenes. In those scenarios, accurate data association between tracked objects and incoming new detections is crucial. This paper presents a tracking system based on the Kalman filter that uses a deep learning approach to the association problem. The proposed architecture consists of three neural networks. First, a convolutional LSTM network extracts spatiotemporal features from a sequence of detections of the same track. Then, a Siamese network calculates the degree of similarity between all tracks and the new detections found at each new frame. Finally, a recurrent LSTM network is used to extract 3D and bounding box information. This model follows the tracking-by-detection paradigm and has been trained with track sequences to be able to handle missed observations and to reduce identity switches. A validation test was carried out on the Argoverse dataset to validate the performance of the proposed system. The developed deep learning approach could improve current multi-object tracking systems based on classic algorithms like the Kalman filter.

Multiple object tracking with behavior detection in crowded scenes using deep learning

Multi-object tracking (MOT) is essential for solving the majority of computer vision issues related to crowd analytics. In an MOT system designing object detection and association are the two main steps. Every frame of the video stream is examined to find the desired objects in the first step. Their trajectories are determined in the second step by comparing the detected objects in the current frame to those in the previous frame. Less missing detections are made possible by an object detection system with high accuracy, which results in fewer segmented tracks. We propose a new deep learning-based model for improving the performance of object detection and object tracking in this research. First, object detection is performed by using the adaptive Mask-RCNN model. After that, the ResNet-50 model is used to extract more reliable and significant features of the objects. Then the effective adaptive feature channel selection method is employed for selecting feature channels to determine the final response map. Finally, an adaptive combination kernel correlation filter is used for multiple object tracking. Extensive experiments were conducted on large object-tracking databases like MOT-20 and KITTI-MOTS. According to the experimental results, the proposed tracker performs better than other cutting-edge trackers when faced with various problems. The experimental simulation is carried out in python. The overall success rate and precision of the proposed algorithm are 95.36% and 93.27%.

3D Multi-Object Tracking Based on Dual-Tracker and D-S Evidence Theory

Most of the current self-driving cars are equipped with a variety of sensors (such as lidar, camera), and it has become a trend to integrate multi-sensor information to achieve 3D multi-object tracking (MOT). In addition, unlike the perfect experimental conditions on public datasets, cars on real roads may experience a 3D sensor failure due to weather, mutual interference, etc. In this paper, we propose a dual-tracker-based 3D MOT system, which fuses 2D and 3D information to track objects and can still ensure good tracking accuracy when the 3D sensor fails in a single frame or multiple consecutive frames. Among them, two internal trackers complete the data association of 3D and 2D information, respectively. However, the outputs of the two internal trackers may conflict. First, we calculate the degree of association of potential matched pairs from the features extracted by each sensor, and then normalize it into the mass function of D-S evidence theory. Finally, we combine the evidence to obtain the final matched pairs of detection instances and track estimates. We do extensive experiments on the KITTI MOT dataset and simulate sensor failure scenarios by ignoring the output of the 3D detector. Using the latest evaluation measure for comparison, the results show that our method outperforms other advanced open-source 3D MOTs in a variety of scenarios.

Fast Timeline Based Multi Object Online Tracking

Abstract Fast state-of-the-art multi-object-tracking (MOT) schemes, such as reported in challenges MOT16 and Mot20, perform tracking on a single sensor, often couple tracking and detection, support only one kind of object representation or don’t take varying latencies and update rates into account. We propose a fast generic MOT system for use in real world applications which is capable of tracking objects from different sensor / detector types with their respective latencies and update rates. An SORT inspired online tracking scheme is extended by time awareness using timelines as unifying concept. The system supports different object, sensor and filter and tracking types by modularizing and generalizing the online tracking scheme, while ensuring high performance using an efficient data-oriented C++-template-based implementation. Using the proposed system we achieve, with comparable evaluation metrics, framerates up to ten times higher than the fastest MOT schemes publicly listed for the axis-aligned bounding-box tracking challenges MOT17 and MOT20.

MOTT: A new model for multi-object tracking based on green learning paradigm

Multi-object tracking (MOT) is one of the most essential and challenging tasks in computer vision (CV). Unlike object detectors, MOT systems nowadays are more complicated and consist of several neural network models. Thus, the balance between the system performance and the runtime is crucial for online scenarios. While some of the works contribute by adding more modules to achieve improvements, we propose a pruned model by leveraging the state-of-the-art Transformer backbone model. Our model saves up to 62% FLOPS compared with other Transformer-based models and almost as twice as fast as them. The results of the proposed model are still competitive among the state-of-the-art methods. Moreover, we will open-source our modified Transformer backbone model for general CV tasks as well as the MOT system.

Multidomain Joint Learning of Pedestrian Detection for Application to Quadrotors

Pedestrian detection and tracking are critical functions in the application of computer vision for autonomous driving in terms of accident avoidance and safety. Extending the application to drones expands the monitoring space from 2D to 3D but complicates the task. Images captured from various angles pose a great challenge for pedestrian detection, because image features from different angles tremendously vary and the detection performance of deep neural networks deteriorates. In this paper, this multiple-angle issue is treated as a multiple-domain problem, and a novel multidomain joint learning (MDJL) method is proposed to train a deep neural network using drone data from multiple domains. Domain-guided dropout, a critical mechanism in MDJL, is developed to self-organize domain-specific features according to neuron impact scores. After training and fine-tuning the network, the accuracy of the obtained model improved in all the domains. In addition, we also combined the MDJL with Markov decision-process trackers to create a multiobject tracking system for flying drones. Experiments are conducted on many benchmarks, and the proposed method is compared with several state-of-the-art methods. Experimental results show that the MDJL effectively tackles many scenarios and significantly improves tracking performance.

A Combined mmWave Tracking and Classification Framework Using a Camera for Labeling and Supervised Learning.

Millimeter wave (mmWave) radar poses prosperous opportunities surrounding multiple-object tracking and sensing as a unified system. One of the most challenging aspects of exploiting sensing opportunities with mmWave radar is the labeling of mmWave data so that, in turn, a respective model can be designed to achieve the desired tracking and sensing goals. The labeling of mmWave datasets usually involves a domain expert manually associating radar frames with key events of interest. This is a laborious means of labeling mmWave data. This paper presents a framework for training a mmWave radar with a camera as a means of labeling the data and supervising the radar model. The methodology presented in this paper is compared and assessed against existing frameworks that aim to achieve a similar goal. The practicality of the proposed framework is demonstrated through experimentation in varying environmental conditions. The proposed framework is applied to design a mmWave multi-object tracking system that is additionally capable of classifying individual human motion patterns, such as running, walking, and falling. The experimental findings demonstrate a reliably trained radar model that uses a camera for labeling and supervision that can consistently produce high classification accuracy across environments beyond those in which the model was trained against. The research presented in this paper provides a foundation for future research in unified tracking and sensing systems by alleviating the labeling and training challenges associated with designing a mmWave classification model.

Detection confidence driven multi-object tracking to recover reliable tracks from unreliable detections

Multi-object tracking (MOT) systems often rely on accurate object detectors; however, accurate detectors are not available in every application domain. We present Robust Confidence Tracking (RCT), an offline MOT algorithm designed for settings where detection quality is poor. Whereas prior methods simply threshold and discard detection confidence information, RCT relies on the exact detection confidence values to increase track quality throughout the entire tracking pipeline. This innovation (along with some simple and well-studied heuristics) allows RCT to achieve robust performance with minimal identity switches, even when provided with completely unfiltered detections. To compare trackers in the presence of unreliable detections, we present a challenging real-world underwater fish tracking dataset, FISHTRAC. In an large-scale evaluation across FISHTRAC, UA-DETRAC, and MOTChallenge data, RCT outperforms a wide variety of trackers, including deep trackers and more classic approaches. We have publically released our FISHTRAC codebase and training dataset at https://github.com/tmandel/fish-detrac, which will facilitate comparing trackers on understudied problems.

One-shot multi-object tracking using CNN-based networks with spatial-channel attention mechanism

Deep learning algorithms for multi-object tracking have made great progress and have powered the emergence of state-of-the-art models to address multi-object tracking problems. Though a lot of efforts have been made, false detections (named “FP”) and missed detections (named “FN”) caused by inaccurate tracking still cannot be well addressed especially in extremely crowded driving situations. To address these problems, we develop a new online one-shot multi-object tracking system based on convolutional neural networks with spatial-channel attention mechanism. Firstly, we propose a feature combination module (FCM) that uses dilated convolution to obtain different receptive fields to adapt to the deformation of the targets, instead of introducing a large number of parameters to deal with the problem of target scale transformation like the recent feature pyramid network. Then, an attention mechanism network (AM-Net) is designed to allow the model to dynamically focus on certain parts of the input that help perform the task and ignore irrelevant information. Finally, we introduce a combination of triple loss and online instance matching loss (TOIM Loss) to distinguish similar instances within a class. Our proposed method is evaluated in three commonly used multi-object tracking datasets including 2DMOT15, MOT16 and MOT20. The results show that our proposed method is generally superior to the compared models.

Design of a Robust System Architecture for Tracking Vehicle on Highway Based on Monocular Camera.

Multi-Target tracking is a central aspect of modeling the environment of autonomous vehicles. A mono camera is a necessary component in the autonomous driving system. One of the biggest advantages of the mono camera is it can give out the type of vehicle and cameras are the only sensors able to interpret 2D information such as road signs or lane markings. Besides this, it has the advantage of estimating the lateral velocity of the moving object. The mono camera is now being used by companies all over the world to build autonomous vehicles. In the expressway scenario, the forward-looking camera can generate a raw picture to extract information from and finally achieve tracking multiple vehicles at the same time. A multi-object tracking system, which is composed of a convolution neural network module, depth estimation module, kinematic state estimation module, data association module, and track management module, is needed. This paper applies the YOLO detection algorithm combined with the depth estimation algorithm, Extend Kalman Filter, and Nearest Neighbor algorithm with a gating trick to build the tracking system. Finally, the tracking system is tested on the vehicle equipped with a forward mono camera, and the results show that the lateral and longitudinal position and velocity can satisfy the need for Adaptive Cruise Control (ACC), Navigation On Pilot (NOP), Auto Emergency Braking (AEB), and other applications.

Multi-object tracking based on attention networks for Smart City system

The Smart City is a hot topic at present. Pedestrian tracking and behavior analysis play an important role in Smart City system. Therefore, it is urgent to design an universal and robust multi-object tracking(MOT) algorithm. MOT is a challenging problem with numerous significant applications, including robot navigation, autonomous driving and pedestrian behavior analysis. Among the existing MOT approaches, MOT models which joint learning detection and Re-ID are attracting increasing attention. Subsequently, there is competition between these two tasks jointly in a network inevitably leading to performance degradation. To address these difficulties, we propose an adaptive joint learning approach, called UnionTrack, for MOT in this paper. It handles with the problem of uniform learning between tasks in online MOT system. Firstly, UnionTrack introduces a scale-aware attention network that aggregates features from different scales for object localization and learning of the corresponding discriminative embeddings. Secondly, a cross-correlation module is designed to construct task-relevant feature maps and interact adaptively during training detection and Re-ID. Then, we perform inter-frame correlation at different stages by using high and low confidence detection to avoid occlusions and backgrounds. UnionTrack is trained in an end-to-end manner to better accommodate the training between detection and Re-ID tasks. Finally, our proposed UnionTrack is compared with state-of-the-art methods in extensively experiments and detailed performance analysis is performed in MOT15, MOT16, MOT17 and MOT20 which are all urban scene to demonstrate the effectiveness of our proposed method. Looking forward to playing a role in the Smart City systems.

MAT: Motion-aware multi-object tracking

Modern multi-object tracking (MOT) systems usually build trajectories through associating per-frame detections. However, facing the challenges of camera motion, fast motion, and occlusion, it is difficult to ensure the quality of long-range tracking or even the tracklet purity, especially for small objects. Most of tracking frameworks depend heavily on the performance of re-identification (ReID) for the data association. Unfortunately, the ReID-based association is not only unreliable and time-consuming, but still cannot address the false negatives for occluded and blurred objects, due to noisy partial-detections, similar appearances, and lack of temporal-spatial constraints. In this paper, we propose an enhanced MOT paradigm, namely Motion-Aware Tracker (MAT). Our MAT is a plug-and-play solution, it mainly focuses on high-performance motion-based prediction, reconnection, and association. First, the nonrigid pedestrian motion and rigid camera motion are blended seamlessly to develop the Integrated Motion Localization (IML) module. Second, the Dynamic Reconnection Context (DRC) module is devised to guarantee the robustness for long-range motion-based reconnection. The core ideas in DRC are the motion-based dynamic-window and cyclic pseudo-observation trajectory filling strategy, which can smoothly fill in the tracking fragments caused by occlusion or blur. At last, we present the 3D Integral Image (3DII) module to efficiently cut off useless track-detection association connections using temporal-spatial constraints. Extensive experiments are conducted on the MOT16&17 challenging benchmarks. The results demonstrate that our MAT can achieve superior performance and surpass other state-of-the-art trackers by a large margin with high efficiency.

Real-Time Multiple Pedestrian Tracking With Joint Detection and Embedding Deep Learning Model for Embedded Systems

This paper proposes an improvement to the multi-object tracking system framework based on the image inputs. By analyzing the role and performance of each block in the original multi-objects tracking system, the blocks of the original system are reconstructed to enhance the efficiency and yield a faster processing speed suiting the real-time applications. In the proposed method, the first two parts of the multi-object tracking system are merged into a single neural network designed for object detection and feature extraction. A new object association judgment method and JDE inspired prediction head are included in order to achieve a better and an outstanding association effect resulting in the overall improvement of the original system by 45.2%. The enhanced method is aimed at the application of smart roadside units and uses fixed-viewpoint image input to achieve multi-object tracking on embedded platforms. The proposed method is implemented on the NVIDIA Jetson AGX Xavier embedded platform. The NVIDIA TensorRT software development kit is used to accelerate the neural network. The overall performance of the proposed system yields better efficiency compared to that of the original SDE design and the overall computing performance achieve up to 14-26 images per second, making it ideal for the real-time smart roadside unit applications.