Multi-object Tracking Algorithm Research Articles

Adaptive multi-object tracking based on sensors fusion with confidence updating

Multi-object tracking (MOT) systems typically rely on object detection results for tracking, so the accuracy of the MOT system is significantly affected by the error of the detector. Changes in error usually lead to unstable tracking. Regarding this problem, we proposed an adaptive MOT method based on detection confidence. At first, we use a simple data fusion method to combine the detection results of LiDAR and camera to reduce the large number of false detections. And then we used a factor based on confidence to adjust the estimating covariance matrix and measurement covariance matrix adaptively. The algorithm can judge which is more reliable between prediction and detection, and choose which is more important in the update step. Meanwhile, we set a factor based on confidence to control the search range in the data association module. Our method reduces the impact of detector error while ensuring accuracy and speed, and improves the robustness of the MOT algorithm. Through experiments conducted on the KITTI multi-object tracking dataset, our method has demonstrated significant advantages over state-of-the-art (SOTA) methods in terms of both accuracy and processing speed. The results of MOTA for 90.02% and FPS for 262.

End-to-End Joint Multi-Object Detection and Tracking for Intelligent Transportation Systems

Environment perception is one of the most critical technology of intelligent transportation systems (ITS). Motion interaction between multiple vehicles in ITS makes it important to perform multi-object tracking (MOT). However, most existing MOT algorithms follow the tracking-by-detection framework, which separates detection and tracking into two independent segments and limit the global efficiency. Recently, a few algorithms have combined feature extraction into one network; however, the tracking portion continues to rely on data association, and requires complex post-processing for life cycle management. Those methods do not combine detection and tracking efficiently. This paper presents a novel network to realize joint multi-object detection and tracking in an end-to-end manner for ITS, named as global correlation network (GCNet). Unlike most object detection methods, GCNet introduces a global correlation layer for regression of absolute size and coordinates of bounding boxes, instead of offsetting predictions. The pipeline of detection and tracking in GCNet is conceptually simple, and does not require complicated tracking strategies such as non-maximum suppression and data association. GCNet was evaluated on a multi-vehicle tracking dataset, UA-DETRAC, demonstrating promising performance compared to state-of-the-art detectors and trackers.

TMTB: Transformer based multi-task branching multi-object tracking algorithm for wide-view scenes

TMTB: Transformer based multi-task branching multi-object tracking algorithm for wide-view scenes

ADVANCEMENTS IN OBJECT DETECTION AND TRACKING ALGORITHMS: AN OVERVIEW OF RECENT PROGRESS

In the realm of computer vision, object detection and tracking constitute fundamental tasks, and recent years have borne witness to astounding advancements attributed to the integration of deep learning techniques. This paper aims to provide a comprehensive overview of the remarkable progress achieved in the domain of object detection and tracking algorithms, shedding light on their profound implications for accuracy, speed, and practical applications in the real world. Advances in object detection have been primarily driven by the adoption of Convolutional Neural Networks (CNNs). Prominent models such as Faster R-CNN, YOLO (You Only Look Once), and SSD (Single Shot Multi-Box Detector) have emerged, significantly enhancing the precision of object identification. Additionally, efficient detectors like Efficient Det and Mobile Net have emerged, striking a balance between accuracy and computational efficiency, thereby enabling real-time applications, even on resource-constrained devices. For tracking, Multi-Object Tracking (MOT) algorithms have undergone improvements, incorporating graph-based approaches such as the Hungarian algorithm and Joint Probabilistic Data Association Filter (JPDAF). These advancements have enabled robust object tracking across video frames. This paper also delves into the synergy between deep learning and real-world applications, emphasizing the impact of these algorithms in domains like autonomous vehicles, surveillance systems, robotics, and augmented reality. KEYWORDS: Object detection, Tracking algorithms, Computer vision, Deep learning, Advancements, Real-world applications, Convolutional Neural Networks, Multi-Object Tracking, Autonomous vehicles, Surveillance, Robotics, Augmented reality.

Intelligent detection and behavior tracking under ammonia nitrogen stress

In this paper, a novel YOLO-based detection model with deformable convolution network (DCN-YOLOv5) is developed, which is concerned with the object and key points detection and behavior tracking problem for Oplegnathus punctatus in the ammonia nitrogen environment. The proposed model can adapt to the posture change of the object by deforming the receptive field, which solves the problem of false and missed detection caused by the movement and occlusion. Moreover, a new multi-object multi-category tracking algorithm (MOMC-Tracking) is proposed to track and plot the trajectory and calculate the key behavioral characteristics parameters. In addition, an executable software which integrates the proposed DCN-YOLOv5 model and the MOMC-Tracking algorithm is proposed. Extensive experiments show that compared with the typical YOLO series of algorithms, the proposed model in this paper performs the best with the highest accuracy and the fastest convergence speed, where the mAP@0.5 and mAP@0.5:0.95 of the proposed DCN-YOLOv5 model are 93.71% and 57.45%, which are respectively improved by 1.78% and 24.77% as compared with those obtained by the original YOLOv5 model.

PrunedYOLO-Tracker: An efficient multi-cows basic behavior recognition and tracking technique

Recognition and tracking of basic cow behaviors in natural environments are critical technologies for cows' health monitoring in smart farming. This study proposes an efficient cow behavior recognition and tracking method (PrunedYOLO-Tracker) to address the main problem of the lack of a real-time and identity-linked approach for analyzing and monitoring cow behaviors. Firstly, the channel pruning algorithm is employed to compress the size and parameters of the base model, YOLO v5l. Secondly, a multi-object tracking (MOT) method, Cascaded-Buffered IoU (C-BIoU), which expands the detection and trajectory matching space by increasing the buffer zone, is proposed to combine the behavior information from detection with the trajectory information from tracking, achieving multi-cows' behavior recognition and tracking. Through experimental verification, the pruned model maintains high detection accuracy while reducing the model size, floating-point operations (FLOPs) and parameters by 73.5%, 76.7% and 74.0%, respectively. Compared to the original model, the pruned model only experiences a slight decrease of 0.2% in F1 score, while achieving a 0.3% increase in mean Average Precision (mAP). In terms of cow tracking performance, when compared to six other multi-object tracking algorithms including DeepSort, DeepMot, OC-Sort, BotSort, StrongSort, and ByteTrack, the proposed method demonstrates the highest High Order Tracking Accuracy (HOTA), Multi-Object Tracking Precision (MOTP) and Identification F1 (IDF1) scores, reaching 72.4%, 86.1% and 80.3%, respectively. The results obtained from testing in multiple cow activity environments demonstrate that the proposed method exhibits excellent performance in behavior recognition and tracking, as well as high-speed processing capabilities, with an average video processing speed of 81 frames per second (FPS). This method possesses the ability to reliably monitor and manage cow behavior in real-time, providing technical support for promptly anomalies detection and cow health status monitoring for dairy farming managers.

A CNN-LSTM Architecture for Marine Vessel Track Association Using Automatic Identification System (AIS) Data.

In marine surveillance, distinguishing between normal and anomalous vessel movement patterns is critical for identifying potential threats in a timely manner. Once detected, it is important to monitor and track these vessels until a necessary intervention occurs. To achieve this, track association algorithms are used, which take sequential observations comprising the geological and motion parameters of the vessels and associate them with respective vessels. The spatial and temporal variations inherent in these sequential observations make the association task challenging for traditional multi-object tracking algorithms. Additionally, the presence of overlapping tracks and missing data can further complicate the trajectory tracking process. To address these challenges, in this study, we approach this tracking task as a multivariate time series problem and introduce a 1D CNN-LSTM architecture-based framework for track association. This special neural network architecture can capture the spatial patterns as well as the long-term temporal relations that exist among the sequential observations. During the training process, it learns and builds the trajectory for each of these underlying vessels. Once trained, the proposed framework takes the marine vessel's location and motion data collected through the automatic identification system (AIS) as input and returns the most likely vessel track as output in real-time. To evaluate the performance of our approach, we utilize an AIS dataset containing observations from 327 vessels traveling in a specific geographic region. We measure the performance of our proposed framework using standard performance metrics such as accuracy, precision, recall, and F1 score. When compared with other competitive neural network architectures, our approach demonstrates a superior tracking performance.

Centralized multi-sensor labeled multi-Bernoulli filter with partially overlapping fields of view

In this paper, the labeled multi-Bernoulli (LMB) filter is extended to the centralized multi-sensor (MS) case, resulting in a multi-sensor multi-object tracking (MS-MOT) algorithm having the ability to cope with partially overlapping fields-of-view (FoVs). Specifically, a joint prediction and update scheme for the MS-LMB filter is presented and the inherent NP-hard multi-dimensional assignment problem is solved via Gibbs sampling. We propose a suboptimal stationary distribution for the Gibbs sampling which admits conditionally independent sampling among multiple sensors thus dramatically decreasing the computational complexity. In addition, in order to tune object initialization when there is no background knowledge of the birth locations, a novel MS measurement-driven birth process is proposed. The proposed birth process has superior ability in terms of suppressing false tracks originating from clutter measurements of multiple sensors. The effectiveness of the proposed algorithm and birth process is demonstrated in simulation experiments concerning MS surveillance scenarios with partially overlapping sensor FoVs.

DOMOPT: A Detection-Based Online Multi-Object Pedestrian Tracking Network for Videos

Due to the problem of low tracking accuracy and weak tracking stability of current multi-object pedestrian tracking algorithms in complex scenes for videos, a Detection-based Online Multi-Object Pedestrian Tracking (DOMOPT) network is proposed. First, a Multi-Level Feature Fusion (MLFF) pedestrian detection network is proposed based on the Center and Scale Prediction (CSP) algorithm. The pyramid convolutional neural network is used as the backbone to enhance the feature extraction capability for small objects. The shallow features and deep features at multiple levels are integrated to fully obtain the position and semantic information to further improve the detection performance for small objects. Then, on the basis of Joint Detection and Embedding (JDE) architecture, a Multi-Branch Pedestrian Appearance (MBPA) feature extraction network is proposed and added into the pedestrian detection network to extract the appearance feature vector corresponding to each pedestrian. The pedestrian appearance feature extraction is treated as a classification task jointly training with the pedestrian detection task, using the multi-task learning strategy. Experimental results show that the proposed network has better tracking accuracy and stability compared with state-of-the-art algorithms.

Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation

Although existing multi-object tracking (MOT) algorithms have obtained competitive performance on various benchmarks, almost all of them train and validate models on the same domain. The domain generalization problem of MOT is hardly studied. To bridge this gap, we first draw the observation that the high-level information contained in natural language is domain invariant to different tracking domains. Based on this observation, we propose to introduce natural language representation into visual MOT models for boosting the domain generalization ability. However, it is infeasible to label every tracking target with a textual description. To tackle this problem, we design two modules, namely visual context prompting (VCP) and visual-language mixing (VLM). Specifically, VCP generates visual prompts based on the input frames. VLM joints the information in the generated visual prompts and the textual prompts from a pre-defined Trackbook to obtain instance-level pseudo textual description, which is domain invariant to different tracking scenes. Through training models on MOT17 and validating them on MOT20, we observe that the pseudo textual descriptions generated by our proposed modules improve the generalization performance of query-based trackers by large margins.

3D reconstruction system and multiobject local tracking algorithm designed for billiards

The use of virtual reality or augmented reality systems in billiards sports are useful tools for pure entertainment or improving the player’s skills. Depending on the purpose of these systems, tracking algorithms based on computer vision must be used. These algorithms are especially useful in systems aiming to reconstruct the trajectories followed by the balls after a strike. However, depending on the billiard modality, the problem of tracking multiple small identical objects, such as balls, is a complex task. In addition, when an amateur or nontop professional player uses low-frame-rate and low-resolution devices, problems such as blurred balls, blurred contours, or fuzzy edges, among others, arise. These effects have a negative impact on ball-tracking accuracy and reconstruction quality. Thus, this work proposes two contributions. The first contribution is a new tracking algorithm called “multiobject local tracking (MOLT)”. This algorithm can track balls with high precision and accuracy even with motion blur caused by low-resolution and low-frame-rate devices. Moreover, the proposed MOLT algorithm is compared with nine tracking methods and four different metrics, outperforming the rest of the methods in the majority of the cases and providing a robust solution. The second contribution is a whole system to track (using the MOLT algorithm) and reconstruct the movements of the balls on a billiard table in a 3D virtual world using computer vision. The proposed system covers all steps from image capture to 3D reconstruction. The 3D reconstruction results have been qualitatively evaluated by different users through a series of questionnaires, obtaining an overall score of 7.6 (out of 10), which indicates that the system is a promising and useful tool for training. Finally, both the MOLT algorithm and the reconstruction system are tested in three billiard modalities: blackball, carom billiards, and snooker.

Vision-Based Real-Time Motion Estimation of Ground Vehicles from Aerial Imagery

This research presents vision-based maneuvering object motion estimation in case of occlusion. Unequal dimension Interactive Multiple Model (UDIMM) approach is applied to increase the motion prediction accuracy when no measurement is available. Current deep learning-based multi-object tracking algorithms cannot track the objects when an occlusion exists, even for quite a short time. The main reason is the motion model used in tracking algorithms. If the measurement coming from the object detector is not available, the prediction model propagates the Kalman Filter’s motion model. In this period, the object may maneuver or accelerate/decelerate so that the tracker cannot assign the tracklets successfully. Eventually, the ID switch occurs, and object tracking gets lost. Using the UDIMM approach, track loss due to the ID switch is highly reduced compared to the simple constant velocity motion model used in the StrongSort multi-object tracker algorithm. A two-dimensional simulation environment is created to compare the ground truth data with the estimation results. Performance results of different filters are compared.

PMOT2023: A Large-Scale Multi-Object Tracking (MOT) Dataset with Application to Phytoplankton Observation

Phytoplankton play a critical role in marine food webs and biogeochemical cycles, and their abundance must be monitored to prevent disasters and improve the marine environment. Although existing algorithms for automatic phytoplankton identification at the image level are available, there are currently no video-level algorithms. This lack of datasets is a significant obstacle to the development of video-level automatic identification algorithms for phytoplankton observations. Deep learning-based algorithms, in particular, require high-quality datasets to achieve optimal results. To address this issue, we propose the PMOT2023 (Phytoplankton Multi-Object Tracking), a multi-video tracking dataset based on 48,000 micrographs captured by in situ observation devices. The dataset comprises 21 classes of phytoplankton and can aid in the development of advanced video-level identification methods. Multi-object tracking algorithms can detect, classify, count, and estimate phytoplankton density. As a video-level automatic identification algorithm, multi-object tracking addresses trajectory tracking, concentration estimation, and other requirements in original phytoplankton observation, helping to prevent marine ecological disasters. Additionally, the PMOT2023 dataset will serve as a benchmark to evaluate the performance of future phytoplankton identification models and provide a foundation for further research on automatic phytoplankton identification algorithms.

Automatic identification and analysis of multi-object cattle rumination based on computer vision.

Rumination in cattle is closely related to their health, which makes the automatic monitoring of rumination an important part of smart pasture operations. However, manual monitoring of cattle rumination is laborious and wearable sensors are often harmful to animals. Thus, we propose a computer vision-based method to automatically identify multi-object cattle rumination, and to calculate the rumination time and number of chews for each cow. The heads of the cattle in the video were initially tracked with a multi-object tracking algorithm, which combined the You Only Look Once (YOLO) algorithm with the kernelized correlation filter (KCF). Images of the head of each cow were saved at a fixed size, and numbered. Then, a rumination recognition algorithm was constructed with parameters obtained using the frame difference method, and rumination time and number of chews were calculated. The rumination recognition algorithm was used to analyze the head image of each cow to automatically detect multi-object cattle rumination. To verify the feasibility of this method, the algorithm was tested on multi-object cattle rumination videos, and the results were compared with the results produced by human observation. The experimental results showed that the average error in rumination time was 5.902% and the average error in the number of chews was 8.126%. The rumination identification and calculation of rumination information only need to be performed by computers automatically with no manual intervention. It could provide a new contactless rumination identification method for multi-cattle, which provided technical support for smart pasture.

YOLO-BYTE: An efficient multi-object tracking algorithm for automatic monitoring of dairy cows

Dairy cows tracking is an essential means to obtain their behavioral information, real-time position, activity data, and health status. A multi-object tracking method (YOLO-BYTE) is proposed to address the problem of missed detection and false detection caused by complex environments in cow individual detection and tracking. The method improves upon the YOLO v7 Backbone network feature extraction module by adding a Self-Attention and Convolution mixed module (ACmix) to account for the uneven spatial distribution and target scale variation of the cows. Additionally, in order to reduce the number of model parameters, an improved lightweight Spatial Pyramid Pooling Cross Stage Partial Connections (SPPCSPC-L) module is adopted to reduce model complexity. At the same time, the state parameters in the Kalman filter are improved by directly predicting the width and height information of the tracking boxes, so as to improve the ByteTrack algorithm to make tracking boxes matching the cows more precisely and accurately. Experimental conducted on the dairy cow object detection and multi-object tracking dataset show that the proposed YOLO-BYTE model achieves a Precision (P) of 97.3% in the dairy cow target detection dataset, with an improved Recall (R) and Average Precision (AP) by 1.1% compared to the original algorithm, and an 18% reduction in model parameters. Moreover, the proposed method demonstrated significant improvements in High Order Tracking Accuracy (HOTA), Multi-Object Tracking Accuracy (MOTA), and Identification F1 (IDF1) by 4.4%, 6.1%, and 3.8%, respectively, compared to the original model, with a decrease of 37.5% in Identity Switch (IDS). The tracker runs in a real-time manner with an average analysis speed of 47 fps. Hence, it is demonstrated that the proposed approach is capable of effective multi-object tracking of dairy cows in natural scenes and provides technical support for non-contact dairy cow automatic monitoring.

Research on Video Synopsis Based on Deep Learning Target Detection and Target Trajectory Optimization

In this era of rapid economic and technological development, monitoring is essential for security issues. However, it is also accompanied by the low efficiency of managing and viewing a large number of surveillance videos every day. Traditional methods are gradually unable to meet the requirements of application, and the video field requires algorithm technology that can better solve practical problems. Intelligent monitoring technology is gradually emerging and developing, based on theories such as computer vision, which can solve many monitoring video problems and greatly improve the efficiency of monitoring video work. Through in-depth exploration of existing video synopsis technology algorithms, this article proposes a new video synopsis method that can effectively detect and track videos, thereby greatly improving the efficiency of storage, transmission, and use of surveillance videos. Experiments have shown that this method can effectively concentrate surveillance videos, and compared with existing methods, it has a better synopsis ratio while ensuring the integrity of video information, effectively reducing collisions between targets, effectively reducing overlap between targets, and achieving good visual effects. Utilizing an improved deep learning object detection and multi object tracking algorithm with added attention mechanism to extract foreground moving targets in videos, and using a mixed Gaussian background modeling algorithm establish a background, laying a stable target foundation for subsequent video concentration. At the same time, design trajectory recombination optimization methods to ensure that the targets do not overlap as much as possible. Reasonably place the targets in the new condensed video sequence, determine the index of all targets in the new condensed video stream, and finally integrate each frame of target images into the background image according to the set index rules, ultimately obtaining the synopsis video.

Benchmarking 2D Multi-Object Detection and Tracking Algorithms in Autonomous Vehicle Driving Scenarios.

Self-driving vehicles must be controlled by navigation algorithms that ensure safe driving for passengers, pedestrians and other vehicle drivers. One of the key factors to achieve this goal is the availability of effective multi-object detection and tracking algorithms, which allow to estimate position, orientation and speed of pedestrians and other vehicles on the road. The experimental analyses conducted so far have not thoroughly evaluated the effectiveness of these methods in road driving scenarios. To this aim, we propose in this paper a benchmark of modern multi-object detection and tracking methods applied to image sequences acquired by a camera installed on board the vehicle, namely, on the videos available in the BDD100K dataset. The proposed experimental framework allows to evaluate 22 different combinations of multi-object detection and tracking methods using metrics that highlight the positive contribution and limitations of each module of the considered algorithms. The analysis of the experimental results points out that the best method currently available is the combination of ConvNext and QDTrack, but also that the multi-object tracking methods applied on road images must be substantially improved. Thanks to our analysis, we conclude that the evaluation metrics should be extended by considering specific aspects of the autonomous driving scenarios, such as multi-class problem formulation and distance from the targets, and that the effectiveness of the methods must be evaluated by simulating the impact of the errors on driving safety.

Detector–Tracker Integration Framework for Autonomous Vehicles Pedestrian Tracking

Pedestrian tracking is an important aspect of autonomous vehicles environment perception in a vehicle running environment. The performance of the existing pedestrian tracking algorithms is limited by the complex traffic environment, the changeable appearance characteristics of pedestrians and the frequent occlusion interaction, which leads to the insufficient accuracy and stability of tracking. Therefore, this paper proposes a detector–tracker integration framework for autonomous vehicle pedestrian tracking. Firstly, a pedestrian objects detector based on the improved YOLOv7 network was established. Space-to-Depth convolution layer was adopted to improve the backbone network of YOLOv7. Then, a novel appearance feature extraction network is proposed, which integrates the convolutional structural re-parameterization idea to construct a full-scale feature extraction block, which is the optimized DeepSORT tracker. Finally, experiments were carried out on MOT17 and MOT20 public datasets and driving video sequences, and the tracking performance of the proposed framework was evaluated by comparing it with the most advanced multi-object tracking algorithms. Quantitative analysis results show that the framework has high tracking accuracy. Compared with DeepSORT, MOTA improves by 2.3% in the MOT17 dataset and MOTA improves by 4.2% in the MOT20 dataset. Through qualitative evaluation on real driving video sequences, the framework proposed in this paper is robust in a variety of climate environments, and can be effectively applied to the pedestrian tracking of autonomous vehicles.

CSMOT: Make One-Shot Multi-Object Tracking in Crowded Scenes Great Again.

The current popular one-shot multi-object tracking (MOT) algorithms are dominated by the joint detection and embedding paradigm, which have high inference speeds and accuracy, but their tracking performance is unstable in crowded scenes. Not only does the detection branch have difficulty in obtaining the accurate object position, but the ambiguous appearance of features extracted by the re-identification (re-ID) branch also leads to identity switches. Focusing on the above problems, this paper proposes a more robust MOT algorithm, named CSMOT, based on FairMOT. First, on the basis of the encoder-decoder network, a coordinate attention module is designed to enhance the information interaction between channels (horizontal and vertical coordinates), which improves its object-detection abilities. Then, an angle-center loss that effectively maximizes intra-class similarity is proposed to optimize the re-ID branch, and the extracted re-ID features are made more discriminative. We further redesign the re-ID feature dimension to balance the detection and re-ID tasks. Finally, a simple and effective data association mechanism is introduced, which associates each detection instead of just the high-score detections during the tracking process. The experimental results show that our one-shot MOT algorithm achieves excellent tracking performance on multiple public datasets and can be effectively applied to crowded scenes. In particular, CSMOT decreases the number of ID switches by 11.8% and 33.8% on the MOT16 and MOT17 test datasets, respectively, compared to the baseline.

A Multi-Object Tracking Algorithm With Center-Based Feature Extraction and Occlusion Handling

For tracking suspicious objects using intelligent robots, Multiple Object Tracking (MOT) has gained great attention. MOT is easily affected by long-term severe occlusion. This work proposes a joint MOT algorithm to handle such occlusion. Pairs of frames in complicated environments are taken as input. A center-based feature extraction framework is designed for precisely detecting objects and extracting their feature maps. A ConvGRU module is applied to learn permanent representations by using historical spatio-temporal information of objects. A Hungarian matching method is applied to match the detected objects and predicted predictions. The proposed algorithm is compared with several representative methods on two public multi-object tracking benchmarks. Furthermore, this work constructs a database with videos captured from street scenarios and uses it to test the proposed algorithm and its peers. Experimental results demonstrate that the proposed algorithm outperforms its peers, especially under long-term severe occlusion, thus advancing the field of MOT.