Multi-object Tracking System Research Articles

Rethinking the Competition Between Detection and ReID in Multiobject Tracking.

Due to balanced accuracy and speed, one-shot models which jointly learn detection and identification embeddings, have drawn great attention in multi-object tracking (MOT). However, the inherent differences and relations between detection and re-identification (ReID) are unconsciously overlooked because of treating them as two isolated tasks in the one-shot tracking paradigm. This leads to inferior performance compared with existing two-stage methods. In this paper, we first dissect the reasoning process for these two tasks, which reveals that the competition between them inevitably would destroy task-dependent representations learning. To tackle this problem, we propose a novel reciprocal network (REN) with a self-relation and cross-relation design so that to impel each branch to better learn task-dependent representations. The proposed model aims to alleviate the deleterious tasks competition, meanwhile improve the cooperation between detection and ReID. Furthermore, we introduce a scale-aware attention network (SAAN) that prevents semantic level misalignment to improve the association capability of ID embeddings. By integrating the two delicately designed networks into a one-shot online MOT system, we construct a strong MOT tracker, namely CSTrack. Our tracker achieves the state-of-the-art performance on MOT16, MOT17 and MOT20 datasets, without other bells and whistles. Moreover, CSTrack is efficient and runs at 16.4 FPS on a single modern GPU, and its lightweight version even runs at 34.6 FPS. The complete code has been released at https://github.com/JudasDie/SOTS.

Efficient Single-Shot Multi-Object Tracking for Vehicles in Traffic Scenarios.

Multi-object tracking is a significant field in computer vision since it provides essential information for video surveillance and analysis. Several different deep learning-based approaches have been developed to improve the performance of multi-object tracking by applying the most accurate and efficient combinations of object detection models and appearance embedding extraction models. However, two-stage methods show a low inference speed since the embedding extraction can only be performed at the end of the object detection. To alleviate this problem, single-shot methods, which simultaneously perform object detection and embedding extraction, have been developed and have drastically improved the inference speed. However, there is a trade-off between accuracy and efficiency. Therefore, this study proposes an enhanced single-shot multi-object tracking system that displays improved accuracy while maintaining a high inference speed. With a strong feature extraction and fusion, the object detection of our model achieves an AP score of 69.93% on the UA-DETRAC dataset and outperforms previous state-of-the-art methods, such as FairMOT and JDE. Based on the improved object detection performance, our multi-object tracking system achieves a MOTA score of 68.5% and a PR-MOTA score of 24.5% on the same dataset, also surpassing the previous state-of-the-art trackers.

Enhancing the association in multi‐object tracking via neighbor graph

Most modern multi-object tracking (MOT) systems for videos follow the tracking-by-detection paradigm, where objects of interest are first located in each frame then associated correspondingly to form their intact trajectories. In this setting, the appearance features of objects usually provide the most important cues for data association, but it is very susceptible to occlusions, illumination variations, and inaccurate detections, thus easily resulting in incorrect trajectories. To address this issue, in this study we propose to make full use of the neighboring information. Our motivations derive from the observations that people tend to move in a group. As such, when an individual target's appearance is remarkably changed, the observer can still identify it with its neighbor context. To model the contextual information from neighbors, we first utilize the spatiotemporal relations among trajectories to efficiently select suitable neighbors for targets. Subsequently, we construct neighbor graph for each target and corresponding neighbors then employ the graph convolutional networks (GCNs) to model their relations and learn the graph features. To the best of our knowledge, it is the first time to explicitly leverage neighbor cues via GCN in MOT. Finally, standardized evaluations on the MOT16 and MOT17 data sets demonstrate that our approach can remarkably reduce the identity switches whilst achieve state-of-the-art overall performance.

A Two-Stage Data Association Approach for 3D Multi-Object Tracking.

Multi-Object Tracking (MOT) is an integral part of any autonomous driving pipelines because it produces trajectories of other moving objects in the scene and predicts their future motion. Thanks to the recent advances in 3D object detection enabled by deep learning, track-by-detection has become the dominant paradigm in 3D MOT. In this paradigm, a MOT system is essentially made of an object detector and a data association algorithm which establishes track-to-detection correspondence. While 3D object detection has been actively researched, association algorithms for 3D MOT has settled at bipartite matching formulated as a Linear Assignment Problem (LAP) and solved by the Hungarian algorithm. In this paper, we adapt a two-stage data association method which was successfully applied to image-based tracking to the 3D setting, thus providing an alternative for data association for 3D MOT. Our method outperforms the baseline using one-stage bipartite matching for data association by achieving 0.587 Average Multi-Object Tracking Accuracy (AMOTA) in NuScenes validation set and 0.365 AMOTA (at level 2) in Waymo test set.

Real-Time Multiobject Tracking Based on Multiway Concurrency.

This paper explored a pragmatic approach to research the real-time performance of a multiway concurrent multiobject tracking (MOT) system. At present, most research has focused on the tracking of single-image sequences, but in practical applications, multiway video streams need to be processed in parallel by MOT systems. There have been few studies on the real-time performance of multiway concurrent MOT systems. In this paper, we proposed a new MOT framework to solve multiway concurrency scenario based on a tracking-by-detection (TBD) model. The new framework mainly focuses on concurrency and real-time based on limited computing and storage resources, while considering the algorithm performance. For the former, three aspects were studied: (1) Expanded width and depth of tracking-by-detection model. In terms of width, the MOT system can support the process of multiway video sequence at the same time; in terms of depth, image collectors and bounding box collectors were introduced to support batch processing. (2) Considering the real-time performance and multiway concurrency ability, we proposed one kind of real-time MOT algorithm based on directly driven detection. (3) Optimization of system level—we also utilized the inference optimization features of NVIDIA TensorRT to accelerate the deep neural network (DNN) in the tracking algorithm. To trade off the performance of the algorithm, a negative sample (false detection sample) filter was designed to ensure tracking accuracy. Meanwhile, the factors that affect the system real-time performance and concurrency were studied. The experiment results showed that our method has a good performance in processing multiple concurrent real-time video streams.

Multiple objects tracking in the UAV system based on hierarchical deep high-resolution network

Robust and high-performance visual multi-object tracking is a big challenge in computer vision, especially in a drone scenario. In this paper, an online Multi-Object Tracking (MOT) approach in the UAV system is proposed to handle small target detections and class imbalance challenges, which integrates the merits of deep high-resolution representation network and data association method in a unified framework. Specifically, while applying tracking-by-detection architecture to our tracking framework, a Hierarchical Deep High-resolution network (HDHNet) is proposed, which encourages the model to handle different types and scales of targets, and extract more effective and comprehensive features during online learning. After that, the extracted features are fed into different prediction networks for interesting targets recognition. Besides, an adjustable fusion loss function is proposed by combining focal loss and GIoU loss to solve the problems of class imbalance and hard samples. During the tracking process, these detection results are applied to an improved DeepSORT MOT algorithm in each frame, which is available to make full use of the target appearance features to match one by one on a practical basis. The experimental results on the VisDrone2019 MOT benchmark show that the proposed UAV MOT system achieves the highest accuracy and the best robustness compared with state-of-the-art methods.

Analysis Based on Recent Deep Learning Approaches Applied in Real-Time Multi-Object Tracking: A Review

The deep learning technique has proven to be effective in the classification and localization of objects on the image or ground plane over time. The strength of the technique’s features has enabled researchers to analyze object trajectories across multiple cameras for online multi-object tracking (MOT) systems. In the past five years, these technical features have gained a reputation in handling several real-time multiple object tracking challenges. This contributed to the increasing number of proposed deep learning methods (DLMs) and networks seen by the computer vision community. The technique efficiently handled various challenges in real-time MOT systems and improved overall tracking performance. However, it experienced difficulties in the detection and tracking of objects in overcrowded scenes and motion variations and confused appearance variations. Therefore, in this paper, we summarize and analyze the 95 contributions made in the past five years on deep learning-based online MOT methods and networks that rank highest in the public benchmark. We review their expedition, performance, advantages, and challenges under different experimental setups and tracking conditions. We also further categorize these methods and networks into four main themes: Online MOT Based Detection Quality and Associations, Real-Time MOT with High-Speed Tracking and Low Computational Costs, Modeling Target Uncertainty in Online MOT, and Deep Convolutional Neural Network (DCNN), Affinity and Data Association. Finally, we discuss the ongoing challenges and directions for future research.

Enhancement Performance of Multiple Objects Detection and Tracking for Real-time and Online Applications

Multi-object detection and tracking systems represent one of the basic and important tasks of surveillance and video traffic systems. Recently. The proposed tracking algorithms focused on the detection mechanism. It showed significant improvement in performance in the field of computer vision. Though. It faced many challenges and problems, such as many blockages and segmentation of paths, in addition to the increasing number of identification keys and false-positive paths. In this work, an algorithm was proposed that integrates information on appearance and visibility features to improve the tracker's performance. It enables us to track multiple objects throughout the video and for a longer period of clogging and buffer a number of ID switches. An effective and accurate data set, tools, and metrics were also used to measure the efficiency of the proposed algorithm. The experimental results show the great improvement in the performance of the tracker, with high accuracy of more than 65%, which achieves competitive performance with the existing algorithms.

A 3D Multiobject Tracking Algorithm of Point Cloud Based on Deep Learning

3D multiobject tracking (MOT) is an important part of road condition detection and hazard warning algorithm in roadside systems and autonomous driving systems. There is a tricky problem in 3D MOT that the identity of occluded object switches after it reappears. Given the good performance of the 2D MOT, this paper proposes a 3D MOT algorithm with deep learning based on the multiobject tracking algorithm. Firstly, a 3D object detector was used to obtain oriented 3D bounding boxes from point clouds. Secondly, a 3D Kalman filter was used for state estimation, and reidentification algorithm was used to match feature similarity. Finally, data association was conducted by combining Hungarian algorithm. Experiments show that the proposed method can still match the original trajectory after the occluded object reappears and run at a rate of 59 FPS, which has achieved advanced results in the existing 3D MOT system.

Experimental Evaluation of a Novel Sensor-Based Sorting Approach Featuring Predictive Real-Time Multiobject Tracking

Sensor-based sorting is a machine vision application that has found industrial application in various fields. An accept-or-reject task is executed by separating a material stream into two fractions. Current systems use line-scanning sensors, which is convenient as the material is perceived during transportation. However, line-scanning sensors yield a single observation of each object and no information about their movement. Due to a delay between localization and separation, assumptions regarding the location and point in time for separation need to be made based on the prior localization. Hence, it is necessary to ensure that all objects are transported at uniform velocities. This is often a complex and costly solution. In this article, we propose a new method for reliably separating particles at nonuniform velocities. The problem is transferred from a mechanical to an algorithmic level. Our novel advanced image processing approach includes equipping the sorter with an area-scan camera in combination with a real-time multiobject tracking system, which enables predictions of the location of individual objects for separation. For the experimental validation of our approach, we present a modular sorting system, which allows comparing sorting results using a line-scan and area-scan camera. Results show that our approach performs reliable separation and hence increases sorting efficiency.

UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking

Effective multi-object tracking (MOT) methods have been developed in recent years for a wide range of applications including visual surveillance and behavior understanding. Existing performance evaluations of MOT methods usually separate the tracking step from the detection step by using one single predefined setting of object detection for comparisons. In this work, we propose a new University at Albany DEtection and TRACking (UA-DETRAC) dataset for comprehensive performance evaluation of MOT systems especially on detectors. The UA-DETRAC benchmark dataset consists of 100 challenging videos captured from real-world traffic scenes (over 140,000 frames with rich annotations, including illumination, vehicle type, occlusion, truncation ratio, and vehicle bounding boxes) for multi-object detection and tracking. We evaluate complete MOT systems constructed from combinations of state-of-the-art object detection and tracking methods. Our analysis shows the complex effects of detection accuracy on MOT system performance. Based on these observations, we propose effective and informative evaluation metrics for MOT systems that consider the effect of object detection for comprehensive performance analysis.

Online-Learning-Based Human Tracking Across Non-Overlapping Cameras

Due to the expanding scale of camera networks, Multiple Camera Tracking (MCT) of humans has received increased attention in recent years. In this paper, we present a novel approach to tracking each human within a single camera and across multiple disjoint cameras. Our framework includes a multi-object tracking and segmentation system, a two-phase feature extractor, and an online-learning-based camera link model estimation. For tracking within a single camera, we apply tracking by segmentation and local object detection with multi-kernel feedback to adaptively improve the robustness of the algorithm. In inter-camera tracking, we introduce an effective integration of appearance and context features. Couples are automatically detected, and the couple feature is also integrated with existing features. The proposed algorithm is scalable by a fully unsupervised online-learning framework. In our experiments, the proposed method outperforms all of the state-of-the-art methods in the benchmark National Laboratory of Pattern Recognition (NLPR)_MCT dataset.

Conditional Random Field (CRF)-Boosting: Constructing a Robust Online Hybrid Boosting Multiple Object Tracker Facilitated by CRF Learning.

Due to the reasonably acceptable performance of state-of-the-art object detectors, tracking-by-detection is a standard strategy for visual multi-object tracking (MOT). In particular, online MOT is more demanding due to its diverse applications in time-critical situations. A main issue of realizing online MOT is how to associate noisy object detection results on a new frame with previously being tracked objects. In this work, we propose a multi-object tracker method called CRF-boosting which utilizes a hybrid data association method based on online hybrid boosting facilitated by a conditional random field (CRF) for establishing online MOT. For data association, learned CRF is used to generate reliable low-level tracklets and then these are used as the input of the hybrid boosting. To do so, while existing data association methods based on boosting algorithms have the necessity of training data having ground truth information to improve robustness, CRF-boosting ensures sufficient robustness without such information due to the synergetic cascaded learning procedure. Further, a hierarchical feature association framework is adopted to further improve MOT accuracy. From experimental results on public datasets, we could conclude that the benefit of proposed hybrid approach compared to the other competitive MOT systems is noticeable.

Multiple human tracking based on distributed collaborative cameras

Due to the horizon limitation of single camera, it is difficult for single camera based multi-object tracking system to track multiple objects accurately. In addition, the possible object occlusion and ambiguous appearances often degrade the performance of single camera based tracking system. In this paper, we propose a new method of multi-object tracking by using multi-camera network. This method can handle many problems in the existing tracking systems, such as partial and total occlusion, ambiguity among objects, time consuming and etc. Experimental results of the prototype of our system on three pedestrian tracking benchmarks demonstrate the effectiveness and practical utility of the proposed method.

Robust multi-object tracking using deep learning framework

Object appearance representation is crucial to any visual tracker. In order to improve the appearance representation, we propose a deep compact and high-level appearance representation applied to a multi-object tracking algorithm, which is called Deep Multi-object Tracking. In this paper, we adopt the deep learning framework to offline obtain generic image features with auxiliary natural images and online fine-tune our Deep Multi-object Tracking system to adapt to appearance changes of the moving objects. Besides, we have fully considered the temporal information denoting the dynamic duration time of each object. Based on the temporal information and particle filter, our Deep Multi-object Tracking algorithm can effectively generate an online learning and updating model to form a discriminative appearance scheme to achieve successful multi-object tracking. Experiments show that the proposed Deep Multi-object Tracking performs well both indoor and outdoor, where the objects undergo large pose, scale, occlusion, and illumination variations in complex scenes including abnormal ones.

Applications for a people detection and tracking algorithm using a time-of-flight camera

This paper outlines a method and applications for detection and tracking of people in depth images, acquired with a low-resolution Time-of-Flight (ToF) camera. This depth sensor is placed perpendicular to the ground in order to provide distance information from a top-view position. Usage of intrinsic and extrinsic camera parameters allows estimation of a ground plane and comparison to the measured distances of the ToF sensor in every pixel. Differences to the expected ground plane define foreground information, that is subsequently combined to associated regions. These regions of interest (ROI) are analyzed to distinguish persons from other objects by using a matched filter that is applied the height segmented depth information of each of these regions. The proposed method separates crowds into individuals and facilitates a multi-object tracking system based on Kalman filtering. Furthermore, we present several applications for the proposed method. Experiments with different crowding situations - from very low to very high density - and different heights of camera placements have proven the applicability and practicability of the system.

Robust online multiobject tracking with data association and track management.

In this paper, we consider a multiobject tracking problem in complex scenes. Unlike batch tracking systems using detections of the entire sequence, we propose a novel online multiobject tracking system in order to build tracks sequentially using online provided detections. To track objects robustly even under frequent occlusions, the proposed system consists of three main parts: 1) visual tracking with a novel data association with a track existence probability by associating online detections with the corresponding tracks under partial occlusions; 2) track management to associate terminated tracks for linking tracks fragmented by long-term occlusions; and 3) online model learning to generate discriminative appearance models for successful associations in other two parts. Experimental results using challenging public data sets show the obvious performance improvement of the proposed system, compared with other state-of-the-art tracking systems. Furthermore, extensive performance analysis of the three main parts demonstrates effects and usefulness of the each component for multiobject tracking.

A video data base system for studying animal behavior.

Classification of farm animal behavior is based on oral or written descriptions of the activity in which the animal is engaged. The quantification of animal behavior for research requires that individuals recognize and code the behavior of the animal under study. The classification of these behaviors can be subjective and may differ among observers. Illustrated guides to animal behavior do not convey the motion associated with most behaviors. Video-based guides offer a method of quantifying behaviors with real-time demonstrations of the components that make up a behavior. An animal behavior encyclopedia has been developed to allow searching and viewing of defined (video-recorded) behaviors on the Internet. This video data base is being developed to initiate a system that automatically extracts animal motion information from an input animal activity video clip using a multiobject tracking and reasoning system. Eventually, the extracted information will be analyzed and described using standard animal behavior definitions (the behavior encyclopedia). The intended applications of the behavior encyclopedia and video tracking system are 1) an accessible data base for defining and illustrating behaviors for both research and teaching and 2) to further automate the collection of animal behavior data.