Multi-object Tracking Task Research Articles

RpTrack: Robust Pig Tracking with Irregular Movement Processing and Behavioral Statistics

Pig behavioral analysis based on multi-object tracking (MOT) technology of surveillance videos is vital for precision livestock farming. To address the challenges posed by uneven lighting scenes and irregular pig movements in the MOT task, we proposed a pig MOT method named RpTrack. Firstly, RpTrack addresses the issue of lost tracking caused by irregular pig movements by using an appropriate Kalman Filter and improved trajectory management. Then, RpTrack utilizes BIoU for the second matching strategy to alleviate the influence of missed detections on the tracking performance. Finally, the method utilizes post-processing on the tracking results to generate behavioral statistics and activity trajectories for each pig. The experimental results under conditions of uneven lighting and irregular pig movements show that RpTrack significantly outperforms four other state-of-the-art MOT methods, including SORT, OC-SORT, ByteTrack, and Bot-SORT, on both public and private datasets. The experimental results demonstrate that RpTrack not only has the best tracking performance but also has high-speed processing capabilities. In conclusion, RpTrack effectively addresses the challenges of uneven scene lighting and irregular pig movements, enabling accurate pig tracking and monitoring of different behaviors, such as eating, standing, and lying. This research supports the advancement and application of intelligent pig farming.

Comparing highly trained handball players’ and non-athletes’ performance in a multi-object tracking task

Highly trained team sports players possess highly developed visual attentional skills, compared with non-athletes. These athletes also have much better motor control than non-athletes. This study compared the performance of intensively trained handball players with non-athletes in a modified version of the multiple object tracking (MOT) task, in which participants were instructed to point at the moving targets with their fingers. We hypothesized that athletes will perform better in the MOT task than non-athletes, and that the increased visual attentional load in the MOT task will affect pointing movements to a smaller degree in athletes than in non-athletes. The results partially support our hypotheses. Highly trained handball players performed much better in the MOT task than non-athletes, which confirms that athletes have better visual attentional skills. Considering the influence of attentional load on motor performance during the MOT task, the results suggest that among athletes, this influence may be present, but limited. However, this result should be interpreted with caution.

PANet: An End-to-end Network Based on Relative Motion for Online Multi-object Tracking

The popular tracking-by-detection paradigm of multi-object tracking (MOT) takes detections of each frame as the input and associates detections from one frame to another. Existing association methods based on the relative motion have attracted attention, because they restrain the effect of noisy detections and improve the performance of MOT. However, these methods depend only on the immediately previous frame, which may easily lead to inaccurate matches and even large accumulated errors. Furthermore, multiple objects involved in occlusions are not fully exploited in these existing methods, which leads to the aggravation of inaccurate matches. Motivated by these issues, we design the pivot to represent each object and propose a novel pivot association network (PANet) for the MOT task. Specifically, pivots are learned from spatial semantic and historical contextual clues, which alleviates the dependency on the immediately previous frame. Our online tracker PANet employs pivots and a lightweight associator to localize tracklets of objects, which can inhibit noise detections and improve the accuracy of tracklet prediction by learning the correlation responses between pivots and spatial search areas. Extensive experiments conducted on two-dimensional MOT15, MOT16, MOT17, and MOT20 demonstrate the effectiveness of the proposed method against numerous state-of-the-art MOT trackers.

An Efficient Method for Monitoring Birds Based on Object Detection and Multi-Object Tracking Networks.

To protect birds, it is crucial to identify their species and determine their population across different regions. However, currently, bird monitoring methods mainly rely on manual techniques, such as point counts conducted by researchers and ornithologists in the field. This method can sometimes be inefficient, prone to errors, and have limitations, which may not always be conducive to bird conservation efforts. In this paper, we propose an efficient method for wetland bird monitoring based on object detection and multi-object tracking networks. First, we construct a manually annotated dataset for bird species detection, annotating the entire body and head of each bird separately, comprising 3737 bird images. We also built a new dataset containing 11,139 complete, individual bird images for the multi-object tracking task. Second, we perform comparative experiments using a state-of-the-art batch of object detection networks, and the results demonstrated that the YOLOv7 network, trained with a dataset labeling the entire body of the bird, was the most effective method. To enhance YOLOv7 performance, we added three GAM modules on the head side of the YOLOv7 to minimize information diffusion and amplify global interaction representations and utilized Alpha-IoU loss to achieve more accurate bounding box regression. The experimental results revealed that the improved method offers greater accuracy, with mAP@0.5 improving to 0.951 and mAP@0.5:0.95 improving to 0.815. Then, we send the detection information to DeepSORT for bird tracking and classification counting. Finally, we use the area counting method to count according to the species of birds to obtain information about flock distribution. The method described in this paper effectively addresses the monitoring challenges in bird conservation.

Inference-Domain Network Evolution: A New Perspective for One-Shot Multi-Object Tracking.

The supervised one-shot multi-object tracking (MOT) algorithms have achieved satisfactory performance benefiting from a large amount of labeled data. However, in real applications, acquiring plenty of laborious manual annotations is not practical. It is necessary to adapt the one-shot MOT model trained on a labeled domain to an unlabeled domain, yet such domain adaptation is a challenging problem. The main reason is that it has to detect and associate multiple moving objects distributed in various spatial locations, but there are obvious discrepancies in style, object identity, quantity, and scale among different domains. Motivated by this, we propose a novel inference-domain network evolution to enhance the generalization ability of the one-shot MOT model. Specifically, we design a spatial topology-based one-shot network (STONet) to perform the one-shot MOT task, where a self-supervision mechanism is employed to stimulate the feature extractor to learn the spatial contexts without any annotated information. Furthermore, a temporal identity aggregation (TIA) module is proposed to assist STONet to weaken the adverse effects of noisy labels in the network evolution. This designed TIA aggregates historical embeddings with the same identity to learn cleaner and more reliable pseudo labels. In the inference domain, the proposed STONet with TIA performs pseudo label collection and parameter update progressively to realize the network evolution from the labeled source domain to an unlabeled inference domain. Extensive experiments and ablation studies conducted on MOT15, MOT17, and MOT20, demonstrate the effectiveness of our proposed model.

Pixel-Guided Association for Multi-Object Tracking.

Propagation and association tasks in Multi-Object Tracking (MOT) play a pivotal role in accurately linking the trajectories of moving objects. Recently, modern deep learning models have been addressing these tasks by introducing fragmented solutions for each different problem such as appearance modeling, motion modeling, and object associations. To bring unification in the MOT task, we introduce a pixel-guided approach to efficiently build the joint-detection and tracking framework for multi-object tracking. Specifically, the up-sampled multi-scale features from consecutive frames are queued to detect the object locations by using a transformer-decoder, and per-pixel distributions are utilized to compute the association matrix according to object queries. Additionally, we introduce a long-term appearance association on track features to learn the long-term association of tracks against detections to compute the similarity matrix. Finally, a similarity matrix is jointly integrated with the Byte-Tracker resulting in a state-of-the-art MOT performance. The experiments with the standard MOT15 and MOT17 benchmarks show that our approach achieves significant tracking performance.

MSA-MOT: Multi-Stage Association for 3D Multimodality Multi-Object Tracking.

Three-dimensional multimodality multi-object tracking has attracted great attention due to the use of complementary information. However, such a framework generally adopts a one-stage association approach, which fails to perform precise matching between detections and tracklets, and, thus, cannot robustly track objects in complex scenes. To address this matching problem caused by one-stage association, we propose a novel multi-stage association method, which consists of a hierarchical matching module and a customized track management module. Specifically, the hierarchical matching module defines the reliability of the objects by associating multimodal detections, and matches detections with trajectories based on the reliability in turn, which increases the utilization of true detections, and, thus, guides accurate association. Then, based on the reliability of the trajectories provided by the matching module, the customized track management module sets maximum missing frames with differences for tracks, which decreases the number of identity switches of the same object and, thus, further improves the association accuracy. By using the proposed multi-stage association method, we develop a tracker called MSA-MOT for the 3D multi-object tracking task, alleviating the inherent matching problem in one-stage association. Extensive experiments are conducted on the challenging KITTI benchmark, and the results show that our tracker outperforms the previous state-of-the-art methods in terms of both accuracy and speed. Moreover, the ablation and exploration analysis results demonstrate the effectiveness of the proposed multi-stage association method.

Multiple Object Tracking of Drone Videos by a Temporal-Association Network with Separated-Tasks Structure

The task of multi-object tracking via deep learning methods for UAV videos has become an important research direction. However, with some current multiple object tracking methods, the relationship between object detection and tracking is not well handled, and decisions on how to make good use of temporal information can affect tracking performance as well. To improve the performance of multi-object tracking, this paper proposes an improved multiple object tracking model based on FairMOT. The proposed model contains a structure to separate the detection and ReID heads to decrease the influence between every function head. Additionally, we develop a temporal embedding structure to strengthen the representational ability of the model. By combing the temporal-association structure and separating different function heads, the model’s performance in object detection and tracking tasks is improved, which has been verified on the VisDrone2019 dataset. Compared with the original method, the proposed model improves MOTA by 4.9% and MOTP by 1.2% and has better tracking performance than the models such as SORT and HDHNet on the UAV video dataset.

PTMOT: A Probabilistic Multiple Object Tracker Enhanced by Tracklet Confidence for Autonomous Driving

Real driving scenarios, due to occlusions and disturbances, provide disordered and noisy measurements, which makes the task of multi-object tracking quite challenging. Conventional approach is to find deterministic data association; however, it has unstable performance in high clutter density. This paper proposes a novel probabilistic tracklet-enhanced multiple object tracker (PTMOT), which integrates Poisson multi-Bernoulli mixture (PMBM) filter with confidence of tracklets. The proposed method is able to realize efficient and robust probabilistic association for 3D multi-object tracking (MOT) and improve the PMBM filter’s continuity by smoothing single target hypothesis with global hypothesis. It consists of two key parts. First, the PMBM tracker based on sets of tracklets is implemented to realize probabilistic fusion of disordered measurements. Second, the confidence of tracklets is smoothed through a smoothing-while-filtering approach. Extensive MOT tests on nuScenes tracking dataset demonstrate that the proposed method achieves superior performance in different modalities.

An Object Point Set Inductive Tracker for Multi-Object Tracking and Segmentation.

Multi-object tracking and segmentation (MOTS) is a derivative task of multi-object tracking (MOT). The new setting encourages the learning of more discriminative high-quality embeddings. In this paper, we focus on the problem of exploring the relationship between the segmenter and the tracker, and propose an efficient Object Point set Inductive Tracker (OPITrack) based on it. First, we discover that after a single attention layer, the high-dimensional, key point embedding will show feature averaging. To alleviate this phenomenon, we propose an embedding generalization training strategy for sparse training and dense testing. This strategy allows the network to increase randomness in training and encourages the tracker to learn more discriminative features. In addition, to learn the desired embedding space, we propose a general Trip-hard sample augmentation loss. The loss uses patches that are not distinguishable by the segmenter to join the feature learning and force the embedding network to learn the difference between false positives and true positives. Our method was validated on two MOTS benchmark datasets and achieved promising results. In addition, our OPITrack can achieve better performance for the raw model while costing less video memory (VRAM) at training time.

SORT-YM: An Algorithm of Multi-Object Tracking with YOLOv4-Tiny and Motion Prediction

Multi-object tracking (MOT) is a significant and widespread research field in image processing and computer vision. The goal of the MOT task consists in predicting the complete tracklets of multiple objects in a video sequence. There are usually many challenges that degrade the performance of the algorithm in the tracking process, such as occlusion and similar objects. However, the existing MOT algorithms based on the tracking-by-detection paradigm struggle to accurately predict the location of the objects that they fail to track in complex scenes, leading to tracking performance decay, such as an increase in the number of ID switches and tracking drifts. To tackle those difficulties, in this study, we design a motion prediction strategy for predicting the location of occluded objects. Since the occluded objects may be legible in earlier frames, we utilize the speed and location of the objects in the past frames to predict the possible location of the occluded objects. In addition, to improve the tracking speed and further enhance the tracking robustness, we utilize efficient YOLOv4-tiny to produce the detections in the proposed algorithm. By using YOLOv4-tiny, the tracking speed of our proposed method improved significantly. The experimental results on two widely used public datasets show that our proposed approach has obvious advantages in tracking accuracy and speed compared with other comparison algorithms. Compared to the Deep SORT baseline, our proposed method has a significant improvement in tracking performance.

Online multi-object tracking using multi-function integration and tracking simulation training

Recently, with the development of deep-learning, the performance of multi-object tracking algorithms based on deep neural networks has been greatly improved. However, most methods separate different functional modules into multiple networks and train them independently on specific tasks. When these network modules are used directly, they are not compatible with each other effectively, nor can they be better adapted to the multi-object tracking task, which leads to a poor tracking effect. Therefore, a network structure is designed to aggregate the regression of objects between frames and the extraction of appearance features into one model to improve the harmony between various functional modules of multi-object tracking. To improve the support for the multi-object tracking task, an end-to-end training method is also proposed to simulate the multi-object tracking process during the training and expand the training data by using the historical position of the target combined with the prediction of the motion model. A metric loss that can take advantage of the historical appearance features of the target is also used to train the extraction module of appearance features to improve the temporal correlation of extracted appearance features. Evaluation results on the MOTChallenge benchmark datasets show that the proposed approach achieves state-of-the-art performance.

CenterTrack3D: Improved CenterTrack More Suitable for Three-Dimensional Objects

Abstract Compared with two-dimensional (2D) multi-object tracking (MOT) algorithms, three-dimensional (3D) multi-object tracking algorithms have more research significance and broad application prospects in the unmanned vehicles research field. Aiming at the problem of 3D multi-object detection and tracking, in this paper, the multi-object tracker CenterTrack, which focuses on 2D multi-object tracking task while ignoring object 3D information, is improved mainly from two aspects of detection and tracking, and the improved network is called CenterTrack3D. In terms of detection, CenterTrack3D uses the idea of attention mechanism to optimize the way that the previous-frame image and the heatmap of previous-frame tracklets are added to the current-frame image as input, and second convolutional layer of the hm output head is replaced by dynamic convolution layer, which further improves the ability to detect occluded objects. In terms of tracking, a cascaded data association algorithm based on 3D Kalman filter is proposed to make full use of the 3D information of objects in the image and increase the robustness of the 3D multi-object tracker. The experimental results show that, compared with the original CenterTrack and the existing 3D multi-object tracking methods, CenterTrack3D achieves 88.75% MOTA for cars and 59.40% MOTA for pedestrians and is very competitive on the KITTI tracking benchmark test set.

Multimodal Multiobject Tracking by Fusing Deep Appearance Features and Motion Information

Multiobject Tracking (MOT) is one of the most important abilities of autonomous driving systems. However, most of the existing MOT methods only use a single sensor, such as a camera, which has the problem of insufficient reliability. In this paper, we propose a novel Multiobject Tracking method by fusing deep appearance features and motion information of objects. In this method, the locations of objects are first determined based on a 2D object detector and a 3D object detector. We use the Nonmaximum Suppression (NMS) algorithm to combine the detection results of the two detectors to ensure the detection accuracy in complex scenes. After that, we use Convolutional Neural Network (CNN) to learn the deep appearance features of objects and employ Kalman Filter to obtain the motion information of objects. Finally, the MOT task is achieved by associating the motion information and deep appearance features. A successful match indicates that the object was tracked successfully. A set of experiments on the KITTI Tracking Benchmark shows that the proposed MOT method can effectively perform the MOT task. The Multiobject Tracking Accuracy (MOTA) is up to 76.40% and the Multiobject Tracking Precision (MOTP) is up to 83.50%.

Transient Signals and Inattentional Blindness in a Multi-object Tracking Task.

Inattentional blindness is a failure to notice an unexpected event when attention is directed elsewhere. The current study examined participants’ awareness of an unexpected object that maintained luminance contrast, switched the luminance once, or repetitively flashed. One hundred twenty participants performed a dynamic tracking task on a computer monitor for which they were instructed to count the number of movement deflections of an attended set of objects while ignoring other objects. On the critical trial, an unexpected cross that did not change its luminance (control condition), switched its luminance once (switch condition), or repetitively flashed (flash condition) traveled across the stimulus display. Participants noticed the unexpected cross more frequently when the luminance feature matched their attention set than when it did not match. Unexpectedly, however, a proportion of the participants who noticed the cross in the switch and flash conditions were statistically comparable. The results suggest that an unexpected object with even a single luminance change can break inattentional blindness in a multi-object tracking task.

A joint appearance model of SRC and MFH for multi-objects tracking

This paper extends sparse representation based classification (SRC) and multi-feature hashing (MFH) into multi-object tracking task, and proposes a joint appearance model of SRC and MFH, which aims at discriminating different objects effectively. Unlike most previous approaches which only focus on producing appearance models for all targets, we further consider discriminative features for distinguishing difficult pairs of targets. Firstly, an SRC based global discriminative appearance model is designed for discriminating all targets. It formulates tracklets association as an SRC problem. A discriminative dictionary learning approach is introduced, which improves the SRC classification performance. By this way, the global discriminative appearance model can distinguish different targets more effectively. Secondly, an MFH based pairwise appearance model is designed. This pairwise appearance model focuses on distinguishable features from two targets without considering other targets or backgrounds, therefore it is more effective for differentiating specific close-by tracklets pairs. Data association framework is employed to generate final tracks. Considerable performance improvements are shown on challenging data sets, particularly in metrics of identity switches.

Exploiting Hierarchical Dense Structures on Hypergraphs for Multi-Object Tracking

Most multi-object tracking algorithms are developed within the tracking-by-detection framework that consider the pairwise appearance similarities between detection responses or tracklets within a limited temporal window, and thus less effective in handling long-term occlusions or distinguishing spatially close targets with similar appearance in crowded scenes. In this work, we propose an algorithm that formulates the multi-object tracking task as one to exploit hierarchical dense structures on an undirected hypergraph constructed based on tracklet affinity. The dense structures indicate a group of vertices that are inter-connected with a set of hyperedges with high affinity values. The appearance and motion similarities among multiple tracklets across the spatio-temporal domain are considered globally by exploiting high-order similarities rather than pairwise ones, thereby facilitating distinguish spatially close targets with similar appearance. In addition, the hierarchical design of the optimization process helps the proposed tracking algorithm handle long-term occlusions robustly. Extensive experiments on various challenging datasets of both multi-pedestrian and multi-face tracking tasks, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.

Block covariance based l1 tracker with a subtle template dictionary

Sparse representation is one of the most influential frameworks for visual tracking. However, when applying this framework to the real-world tracking applications, there are still many challenges such as appearance variations and background noise. In this paper, we propose a new l1-regularized sparse representation based tracking algorithm. The contributions of our work are: (1) A block-division based covariance feature is incorporated into the sparse representation framework. This feature has two advantages—(a) the feature is more discriminative than the original image patch and (b) the block information is robust for occlusion reasoning. (2) A subtle template dictionary is constructed including a fixed template, a stable template and other variational templates; and these templates are selectively updated to capture the appearance variations and prevent the model from drifting. (3) The sparse representation framework is extended to multi-object tracking, where the multi-object tracking task can be easily decentralized to a set of individual trackers. Experimental results demonstrate that, compared with several state-of-the-art tracking algorithms, the proposed algorithm is more robust and effective.

A dynamic neural field model of multi-object tracking

The Multi-Object Tracking (MOT) task has been a central tool for studying object-based attention and the nature of the connection between on-line percepts and the location of objects in a scene. Despite a long empirical tradition, there are few formal models that specify the processes that underlie performance in this dynamic task. Here we propose a Dynamic Neural Field (DNF) model that captures the central empirical effects in this literature and integrates multi-object tracking with other known characteristics of the visuo-spatial cognitive system. Our DNF model has three layers of spatially-tuned neurons. The first layer—the perceptual layer—receives input from the world and forms activation “peaks” that are stabilized by input. The second layer—the working memory layer—receives both direct input from the world as well as input from the perceptual layer. This field is able to form self-sustaining peaks that maintain activation even when input is removed. Finally, both the perceptual and working memory layers are reciprocally connected to a shared layer of inhibitory interneurons. To explain the model's performance, consider a single trial that begins with the presentation of 6 objects, 3 of which are cued. In response to this input, the DNF model forms 6 stabilized peaks of activation in the perceptual layer and three working memory peaks due to the strong input at three locations. These activation peaks are maintained by locally-excitatory and laterally-inhibitory interactions. Consequently, once the inputs begin moving the activation peaks in working memory track their associated inputs provided that the inputs are within the range of locally-excitatory interactions. Moreover, the peaks maintain their fidelity via lateral inhibition even when they come near other peaks. We will show how this basic architecture can perform the MOT task and show proximity effects as well as the ability to track objects behind occluders.