Multi-object Tracking Method Research Articles

Segment as Points for Efficient and Effective Online Multi-Object Tracking and Segmentation.

Current multi-object tracking and segmentation (MOTS) methods follow the tracking-by-detection paradigm and adopt 2D or 3D convolutions to extract instance embeddings for instance association. However, due to the large receptive field of deep convolutional neural networks, the foreground areas of the current instance and the surrounding areas containing the nearby instances or environments are usually mixed up in the learned instance embeddings, resulting in ambiguities in tracking. In this paper, we propose a highly effective method for learning instance embeddings based on segments by converting the compact image representation to un-ordered 2D point cloud representation. In this way, the non-overlapping nature of instance segments can be fully exploited by strictly separating the foreground point cloud and the background point cloud. Moreover, multiple informative data modalities are formulated as point-wise representations to enrich point-wise features. For each instance, the embedding is learned on the foreground 2D point cloud, the environment 2D point cloud, and the smallest circumscribed bounding box. Then, similarities between instance embeddings are measured for the inter-frame association. In addition, to enable the practical utility of MOTS, we modify the one-stage instance segmentation method SpatialEmbedding for instance segmentation. The resulting efficient and effective framework, named PointTrackV2, outperforms all the state-of-the-art methods including 3D tracking methods by large margins (4.8 percent higher sMOTSA for pedestrians over MOTSFusion) with the near real-time speed (20 FPS evaluated on a single 2080Ti). Extensive evaluations on three datasets demonstrate both the effectiveness and efficiency of our method. Furthermore, as crowded scenes for cars are insufficient in current MOTS datasets, we provide a more challenging dataset named APOLLO MOTS with a much higher instance density.

Multi-object tracking using Deep SORT and modified CenterNet in cotton seedling counting

Accurate plant density information is important for crop yield and quality. In general, human has to estimate plant density either in field or with accessory equipment, which is time-consuming and inaccurate. In this work, multi-object tracking method based on tracking-by-detection strategy was developed to automatically count cotton seedlings. Videos were collected 0.5 m above cotton seedlings, and analyzed to train object detection model and evaluate counting accuracy with a separate dataset (TAMU2015-ID). An advanced anchor-free object detection model was developed using CenterNet to detect cotton seedling and extract its identity embedding. The localization and identity information were fused based on Deep SORT for data association. The object detection model outperformed Faster R-CNN model with an F1 score of 0.982 (IOU0.5) and 0.937 (IOU0.8), and an average precision of 0.9901 (IOU0.5) and 0.8998 (IOU0.8). The counting results were fitted to ground truth with a R2 of 0.967 and RMSE of 0.394. We evaluated the method on TAMU2015-ID to get a R2 of 0.99 and RMSE of 0.8.

Wildlife 3D multi-object tracking

The study of wildlife populations and species has gained increased relevance due to significant endangerment, loss of habitats and world climate change. Using camera traps for monitoring is a common method to estimate species diversity, occupancy and relative abundance. We present a 3D multi-object tracking method for wildlife camera trapping using RGB-D cameras. We infer precise three-dimensional pose estimations of individual animals (i.e. locations and orientations) and also three-dimensional estimations of their observed movements, including velocity estimations. More precisely, we achieved a scaled Average Multi-Object Tracking Accuracy (sAMOTA) of 88.43%. The code is available athttps://github.com/m-klasen/3D_wildlife_tracking.

DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association

In the recent literature, on the one hand, many 3D multi-object tracking (MOT) works have focused on tracking accuracy and neglected computation speed, commonly by designing rather complex cost functions and feature extractors. On the other hand, some methods have focused too much on computation speed at the expense of tracking accuracy. In view of these issues, this paper proposes a robust and fast camera-LiDAR fusion-based MOT method that achieves a good trade-off between accuracy and speed. Relying on the characteristics of camera and LiDAR sensors, an effective deep association mechanism is designed and embedded in the proposed MOT method. This association mechanism realizes tracking of an object in a 2D domain when the object is far away and only detected by the camera, and updating of the 2D trajectory with 3D information obtained when the object appears in the LiDAR field of view to achieve a smooth fusion of 2D and 3D trajectories. Extensive experiments based on the typical datasets indicate that our proposed method presents obvious advantages over the state-of-the-art MOT methods in terms of both tracking accuracy and processing speed. Our code is made publicly available for the benefit of the community.

Resource-Efficient Visual Multiobject Tracking on Embedded Device

Multi-object tracking (MOT) is a crucial technology for security surveillance, which is computationally intensive due to the requirement of processing a large number of video streams within low latency in practice. The input video streams of MOT are processed on a cloud computing center with abundant computational capability, posing heavy pressures on delivering video streams to the cloud. Recent advances in the Internet of Things (IoT) technology provide edge-computing-based solutions for video analytics at scale. However, the gap between MOT’s high computational capability demand and IoT devices’ resource-constrained nature remains significant. In this paper, a resource-efficient multi-object tracking method (REMOT) is proposed for real-time surveillance on IoT embedded devices, including an affinity measurement based on an appearance model with angular triplet loss and a motion association that substitutes the time-consuming graph-based data association stage. Considering the trade-off between latency and accuracy, we design an optimization strategy on the parallel processing of deep learning models’ layers to accelerate the inference speed with less accuracy loss. Besides, we employ a model compression strategy for model size reduction. Experiments on MOT16 and MOT17 benchmarks demonstrate that REMOT reduces 2.4x latency compared with the original implementation and achieves a running speed of 81 frames per second (fps) on an embedded device with only a marginal accuracy loss (6%), which meets the requirements of real-time processing and low-latency response for surveillance.

Improving visual multi‐object tracking algorithm via integrating GM‐PHD and correlation filter

The traditional visual multi-object tracking methods based on the Gaussian mixture probability hypothesis density filter are generally not well adapted for tracking the targets in the complex scenarios, where there are a large number of unknowable newborn objects and occluded objects, even some missing objects cannot be associated with their previous trajectories when they are redetected. An improved visual multi-object tracking algorithm is proposed by integrating an improved efficient convolution operator of the correlation filter and the Gaussian mixture probability hypothesis density filter. First, a similarity matrix based on the intersection-of-union is proposed for classifying the objects of survival objects, newborn objects, and then the improved efficient convolution operator method is employed to further identify whether the objects disappear or are missing. Moreover, the feature pyramid similarity is proposed to update the objects for enhancing the tracking accuracy. Finally, compared with some challenging methods on some challenging video sequences from publicly available MOT17 dataset, the proposed Gaussian mixture probability hypothesis density–feature pyramid similarity—efficient convolution operator* method has a good performance on detecting the newborn objects, occluded objects, blurring objects and re-identifying the missing objects with higher multiple object tracking accuracy.

360^{circ } real-time and power-efficient 3D DAMOT for autonomous driving applications

Autonomous Driving (AD) promises an efficient, comfortable and safe driving experience. Nevertheless, fatalities involving vehicles equipped with Automated Driving Systems (ADSs) are on the rise, especially those related to the perception module of the vehicle. This paper presents a real-time and power-efficient 3D Multi-Object Detection and Tracking (DAMOT) method proposed for Intelligent Vehicles (IV) applications, allowing the vehicle to track 360^{circ } surrounding objects as a preliminary stage to perform trajectory forecasting to prevent collisions and anticipate the ego-vehicle to future traffic scenarios. First, we present our DAMOT pipeline based on Fast Encoders for object detection and a combination of a 3D Kalman Filter and Hungarian Algorithm, used for state estimation and data association respectively. We extend our previous work ellaborating a preliminary version of sensor fusion based DAMOT, merging the extracted features by a Convolutional Neural Network (CNN) using camera information for long-term re-identification and obstacles retrieved by the 3D object detector. Both pipelines exploit the concepts of lightweight Linux containers using the Docker approach to provide the system with isolation, flexibility and portability, and standard communication in robotics using the Robot Operating System (ROS). Second, both pipelines are validated using the recently proposed KITTI-3DMOT evaluation tool that demonstrates the full strength of 3D localization and tracking of a MOT system. Finally, the most efficient architecture is validated in some interesting traffic scenarios implemented in the CARLA (Car Learning to Act) open-source driving simulator and in our real-world autonomous electric car using the NVIDIA AGX Xavier, an AI embedded system for autonomous machines, studying its performance in a controlled but realistic urban environment with real-time execution (results).

Online multi-object tracking with unsupervised re-identification learning and occlusion estimation

Occlusion between different objects is a typical challenge in Multi-Object Tracking (MOT), which often leads to inferior tracking results due to the missing detected objects. The common practice in multi-object tracking is re-identifying the missed objects after their reappearance. Though tracking performance can be boosted by the re-identification, the annotation of identity is required to train the model. In addition, such practice of re-identification still can not track those highly occluded objects when they are missed by the detector. In this paper, we focus on online multi-object tracking and design two novel modules, the unsupervised re-identification learning module and the occlusion estimation module, to handle these problems. Specifically, the proposed unsupervised re-identification learning module does not require any (pseudo) identity information nor suffer from the scalability issue. The proposed occlusion estimation module tries to predict the locations where occlusions happen, which are used to estimate the positions of missed objects by the detector. Our study shows that, when applied to state-of-the-art MOT methods, the proposed unsupervised re-identification learning is comparable to supervised re-identification learning, and the tracking performance is further improved by the proposed occlusion estimation module.

Multi-Pedestrian Tracking in Crowded Scenes by Modeling Movement Behavior and Optimizing Kalman Filter

Most multi-object tracking methods have achieved good results in tracking multiple pedestrians with Kalman filter, but their tracking performance in crowded scenes is still poor due to pedestrian avoidance and frequent occlusion. In crowded scenes, the pedestrian trajectory prediction with Kalman filter alone is unreliable. In this paper, a two-dimensional field-of-view avoidance force model (AFM) is proposed to assist the Kalman filter prediction by sensing the avoidance force and then complete pedestrian tracking. In the model, each pedestrian has a two-dimensional field of view to perceive the avoidance force, which determines the next predicted trajectory. In real scenes, pedestrians tend to be more concerned about the surrounding area, so different areas are set to simulate the attention mechanisms of pedestrians in real scenes. In the FairMOT model, AFM is used to optimize the pedestrian state values of Kalman filter prediction and the optimized model is trained on the MOT16 public dataset. The experimental results on the MOT20 benchmark dataset show that compared with the mainstream tracking model FairMOT, our method respectively improves MOTA by 2.7% and IDF1 by 2.2%. Our method also achieves the good performance on MOT15, MOT16, and MOT17 tracking benchmarks.

Multi-Object Tracking and Segmentation With Embedding Mask-Based Affinity Fusion in Hierarchical Data Association

In this paper, we propose a highly feasible fully online multi-object tracking and segmentation (MOTS) method that uses instance segmentation results as an input. The proposed method is based on the Gaussian mixture probability hypothesis density (GMPHD) filter, a hierarchical data association (HDA), and a mask-based affinity fusion (MAF) model to achieve high-performance online tracking. The HDA consists of two associations: segment-to-track and track-to-track associations. One affinity, for position and motion, is computed by using the GMPHD filter, and the other affinity, for appearance is computed by using the responses from single object trackers such as kernalized correlation filter, SiamRPN, and DaSiamRPN. These two affinities are simply fused by using a score-level fusion method such as min-max normalization referred to as MAF. In addition, to reduce the number of false positive segments, we adopt mask IoU-based merging (mask merging). The proposed MOTS framework with the key modules: HDA, MAF, and mask merging, is easily extensible to simultaneously track multiple types of objects with CPU-only execution in parallel processing. In addition, the developed framework only requires simple parameter tuning unlike many existing MOTS methods that need intensive hyperparameter optimization. In the experiments on the two popular MOTS datasets, the key modules show some improvements. For instance, ID-switch decreases by more than half compared to a baseline method in the training sets. In conclusion, our tracker achieves state-of-the-art MOTS performance in the test sets.

A method for joint detection and re-identification in multi-object tracking

In order to better balance the detection accuracy and tracking speed, we propose an online balanced multi-object tracking method (BalMOT), which integrates object detection and appearance extraction into a single network, and can simultaneously output detection and appearance embedding. We also model the training of classification, regression, and embedding features as a multi-task training problem and each part is weighted based on the task-independent uncertainty method. In addition, we introduce the transition layer to optimize the repeated gradient information in the network and reduce the training cost. Through the training, our BalMOT system reaches 71.9% multiple object tracking accuracy (MOTA) on the MOT17 challenge dataset, and the speed fluctuates between 17.4 ~ 22.3 frames per second (FPS) according to the size of the input image.

Traffic incident detection based on a global trajectory spatiotemporal map

Traffic incidents endanger the smooth running of vehicles. Congestion caused by traffic incidents has caused a waste of time and fuel and seriously affected transportation efficiency. At present, most methods use manual judgment or image features to detect traffic incidents, but these methods lack timeliness, leading to secondary incidents. For dangerous road sections such as ramp-free and long downhills, this paper proposes an algorithm to quickly detect traffic incidents based on a spatiotemporal map of vehicle trajectories. First, a vehicle dataset from the monitoring perspective is constructed, and an improved YOLOv4 detection algorithm is used to detect images organized as batches. Based on the detection result, the multi-object tracking method of vehicle speed prediction in key frames is used to obtain the vehicle trajectory. Then according to the vehicle trajectory obtained in a single scene, the vehicle trajectory is reidentified and associated in the continuous monitoring scene to construct a long-distance vehicle trajectory spatiotemporal map. Finally, according to the distribution and generation status of the trajectory in the spatiotemporal map, traffic incidents such as vehicle parking, vehicle speeding, and vehicle congestion are analyzed. Experimental results show that the proposed method greatly increases the speed of vehicle detection and tracking and obtains high mAP, MOTA, and MOTP indicators. The global spatiotemporal map constructed by trajectory reidentification can achieve high detection rates for traffic incidents, reduce the average elapsed time, and avoid the problems of the inaccuracy of analyzing image features.

Online multi‐object tracking based on time and frequency domain features

Multi-object tracking (MOT) can be considered as an interesting field in computer vision research. Its application can be found in video motion analysis, smart interfaces, and visual surveillance. It is a challenging issue due to difficulties made by a variable number of objects and interaction between them. In this work, a new method for online MOT based on time and frequency domain features is presented. The features are obtained from the wavelet transform and fractal dimension. The modified cuckoo optimization algorithm is utilized for feature selection, which has the ability such as fast convergence and global optima finding. The features are given for learning vector quantization, which is a supervised artificial neural network (ANN). It is used to classify the dataset. To evaluate the performance of the presented technique, simulations are performed using the ETH Mobile Platform and VS-PETS 2009 datasets. The simulation results show the superiority of the presented technique for MOT compared to earlier studies in terms of accuracy. The mostly tracked values for the datasets are 74.3% and 97.2%, which leads to at least 4.2% and 2.5% better performance according to the other methods, respectively.

Online Multiple Object Tracking Using a Novel Discriminative Module for Autonomous Driving

Multi object tracking (MOT) is a key research technology in the environment sensing system of automatic driving, which is very important to driving safety. Online multi object tracking needs to accurately extend the trajectory of multiple objects without using future frame information, so it will face greater challenges. Most of the existing online MOT methods are anchor-based detectors, which have many misdetections and missed detection problems, and have a poor effect on the trajectory extension of adjacent object objects when they are occluded and overlapped. In this paper, we propose a discrimination learning online tracker that can effectively solve the occlusion problem based on an anchor-free detector. This method uses the different weight characteristics of the object when the occlusion occurs and realizes the extension of the competition trajectory through the discrimination module to prevent the ID-switch problem. In the experimental part, we compared the algorithm with other trackers on two public benchmark datasets, MOT16 and MOT17, and proved that our algorithm has achieved state-of-the-art performance, and conducted a qualitative analysis on the convincing autonomous driving dataset KITTI.

Characterization of microscopic deformation of materials using deep learning algorithms

Microstructure-informed design approach is set to revolutionize the design of metals and alloy components for aerospace applications. In this approach, a designer utilizes the influence of individual microstructural features on microscopic deformation to yield desirable macroscopic properties. Therefore, the development of advanced experimental capabilities that enable detailed characterization of microscopic deformation of material test specimens is critical to realize this paradigm shift in practice. However, extracting the complex characteristics of microscopic deformation hidden in raw image data is quite challenging. In this article, we propose an automated data extraction and analysis method based on instance segmentation and tracking of microstructural features using deep learning (DL) and image processing algorithms. The method consists of a trained mask Region-based Convolutional Neural Network (mask R-CNN) DL model combined with a regional instance segmentation approach for the instance segmentation of features, an intersection over union based multi-object tracking method to track segmented instances as they deform, and kinematics models to extract the material characteristics from the geometrical data of the deforming instances. The method is then validated by characterizing the microscopic deformation of an additively manufactured 316L stainless steel coupon specimen under quasi-static tensile loading. Our study presents a general framework for advancing deep learning algorithms to solve complex problems in the field of experimental mechanics.

Multiple Pedestrians and Vehicles Tracking in Aerial Imagery Using a Convolutional Neural Network

In this paper, we address various challenges in multi-pedestrian and vehicle tracking in high-resolution aerial imagery by intensive evaluation of a number of traditional and Deep Learning based Single- and Multi-Object Tracking methods. We also describe our proposed Deep Learning based Multi-Object Tracking method AerialMPTNet that fuses appearance, temporal, and graphical information using a Siamese Neural Network, a Long Short-Term Memory, and a Graph Convolutional Neural Network module for more accurate and stable tracking. Moreover, we investigate the influence of the Squeeze-and-Excitation layers and Online Hard Example Mining on the performance of AerialMPTNet. To the best of our knowledge, we are the first to use these two for regression-based Multi-Object Tracking. Additionally, we studied and compared the L1 and Huber loss functions. In our experiments, we extensively evaluate AerialMPTNet on three aerial Multi-Object Tracking datasets, namely AerialMPT and KIT AIS pedestrian and vehicle datasets. Qualitative and quantitative results show that AerialMPTNet outperforms all previous methods for the pedestrian datasets and achieves competitive results for the vehicle dataset. In addition, Long Short-Term Memory and Graph Convolutional Neural Network modules enhance the tracking performance. Moreover, using Squeeze-and-Excitation and Online Hard Example Mining significantly helps for some cases while degrades the results for other cases. In addition, according to the results, L1 yields better results with respect to Huber loss for most of the scenarios. The presented results provide a deep insight into challenges and opportunities of the aerial Multi-Object Tracking domain, paving the way for future research.

Multi-object Tracking Method Based on Efficient Channel Attention and Switchable Atrous Convolution

In recent years,object detection and data association have getting remarkable progress which are the core components for multi-object tracking. In multi-object tracking field,the main strategy is tracking-by-detection. Although the detection based tracking method can get great results, it is relies on the performance of the detector. In complex scene, detector can not provide reliable results. Moreover,due to the incorrect detection results, data association process can not be trusted. Based on this motivation, this paper focuses on improving the accuracy of detection and data association. We introduce the efficient channel attention module to the backbone network, which can adaptively extract important information in images. Furthermore, we apply switchable atrous convolution in the network to dynamically adjust the receptive field according to object changes. In data association process, the appearance features with minimum occlusion are saved for each existing trajectory, which are used for re-associate after the objects are lost. Extensive experiments on MOT16,MOT17 and MOT20 challenging datasets demonstrate that our method is comparable with the state-of-the-art multi-object tracking methods.

Learning for Graph Matching based Multi-object Tracking in Auto Driving

Multi-object tracking in autonomous driving aims to represent trajectories of moving objects for planning system of the vehicle. In this paper, we propose a new tracking-bydetection scheme based on deep neural networks for multi-object detection and tracking in autonomous driving scene. We first introduce a light-weight neural network branch for fast object detection, and based on the detection results on each frame, we build two object graphs for consecutive frames separately, where the vertices of the graph represent the objects in the image, and the edges of the graph represent the spatial relations between the objects. We then formulate the multi-object tracking problem as the graph matching process by learning the relevance between objects from another object association network branch. Experiments results on the MOT multi-object tracking dataset show that the proposed object detection and tracking approach achieves comparable results with state-of-the-art deep learning based multi-object tracking methods, and outperforms them in tracking efficiency, which ensures real-time multiobject tracking for autonomous driving.

CenterTrack3D: Improved CenterTrack More Suitable for Three-Dimensional Objects

Abstract Compared with two-dimensional (2D) multi-object tracking (MOT) algorithms, three-dimensional (3D) multi-object tracking algorithms have more research significance and broad application prospects in the unmanned vehicles research field. Aiming at the problem of 3D multi-object detection and tracking, in this paper, the multi-object tracker CenterTrack, which focuses on 2D multi-object tracking task while ignoring object 3D information, is improved mainly from two aspects of detection and tracking, and the improved network is called CenterTrack3D. In terms of detection, CenterTrack3D uses the idea of attention mechanism to optimize the way that the previous-frame image and the heatmap of previous-frame tracklets are added to the current-frame image as input, and second convolutional layer of the hm output head is replaced by dynamic convolution layer, which further improves the ability to detect occluded objects. In terms of tracking, a cascaded data association algorithm based on 3D Kalman filter is proposed to make full use of the 3D information of objects in the image and increase the robustness of the 3D multi-object tracker. The experimental results show that, compared with the original CenterTrack and the existing 3D multi-object tracking methods, CenterTrack3D achieves 88.75% MOTA for cars and 59.40% MOTA for pedestrians and is very competitive on the KITTI tracking benchmark test set.

Relation3DMOT: Exploiting Deep Affinity for 3D Multi-Object Tracking from View Aggregation.

Autonomous systems need to localize and track surrounding objects in 3D space for safe motion planning. As a result, 3D multi-object tracking (MOT) plays a vital role in autonomous navigation. Most MOT methods use a tracking-by-detection pipeline, which includes both the object detection and data association tasks. However, many approaches detect objects in 2D RGB sequences for tracking, which lacks reliability when localizing objects in 3D space. Furthermore, it is still challenging to learn discriminative features for temporally consistent detection in different frames, and the affinity matrix is typically learned from independent object features without considering the feature interaction between detected objects in the different frames. To settle these problems, we first employ a joint feature extractor to fuse the appearance feature and the motion feature captured from 2D RGB images and 3D point clouds, and then we propose a novel convolutional operation, named RelationConv, to better exploit the correlation between each pair of objects in the adjacent frames and learn a deep affinity matrix for further data association. We finally provide extensive evaluation to reveal that our proposed model achieves state-of-the-art performance on the KITTI tracking benchmark.