Multiple Object Tracking Research Articles

AdapTrack: An adaptive FairMOT tracking method applicable to marine ship targets

Ship tracking at sea is faced with the disadvantages of complex sea conditions and the large influence of ship occlusion on the tracker. Therefore, we propose a method called AdapTrack based on On the Fairness of Detection and Re-Identification in Multiple Object Tracking (FairMOT) which is suitable for marine targets. The search strategy of trivial augmentation is used to randomly select suitable data augmentation methods and strengths. Then, based on the FairMOT tracking framework, we change the sampling selection method of positive and negative samples from a two-dimensional Gaussian distribution with the same variances to a two-dimensional Gaussian distribution with different variances. It is limited by the bounding box (bbox) of ground truth. This method can improve the detection algorithm’s fitness to the ship target. At the same time, we use Multi-Object Tracking by Associating Every Detection Box (ByteTrack)’s double-threshold strategy to divide detection bboxes, which improves the matching and inference speed. In the first stage of data association, the high-scoring bbox calculates the cost matrix of data association through the Re-identification (Re-ID) model. In the second stage, the Intersection over Union(IOU) cost matrix is calculated after merging low-scoring detection bboxes and unmatched detection bboxes of the first stage. The method achieves Multiple Object Tracking Accuracy (MOTA) of 36.1, Identification F-Score (IDF1) of 47.3, and Frames Per Second (FPS) of 28.79 on the Singapore-Marine-dataset. Experiments show that this method can better alleviate Identification Switch (ID-switch) and ensure the real-time tracking of complex and changeable ship target tracking at sea.

Study Analysis on Tracking Multiple Objects in Presence of Inter Occlusion in Unmanned LC

Study Analysis on Tracking Multiple Objects in Presence of Inter Occlusion in Unmanned LC

Neural α Oscillations and Pupil Size Differentially Index Cognitive Demand under Competing Audiovisual Task Conditions.

Cognitive demand is thought to modulate two often used, but rarely combined, measures: pupil size and neural α (8-12 Hz) oscillatory power. However, it is unclear whether these two measures capture cognitive demand in a similar way under complex audiovisual-task conditions. Here we recorded pupil size and neural α power (using electroencephalography), while human participants of both sexes concurrently performed a visual multiple object-tracking task and an auditory gap detection task. Difficulties of the two tasks were manipulated independent of each other. Participants' performance decreased in accuracy and speed with increasing cognitive demand. Pupil size increased with increasing difficulty for both the auditory and the visual task. In contrast, α power showed diverging neural dynamics: parietal α power decreased with increasing difficulty in the visual task, but not with increasing difficulty in the auditory task. Furthermore, independent of task difficulty, within-participant trial-by-trial fluctuations in pupil size were negatively correlated with α power. Difficulty-induced changes in pupil size and α power, however, did not correlate, which is consistent with their different cognitive-demand sensitivities. Overall, the current study demonstrates that the dynamics of the neurophysiological indices of cognitive demand and associated effort are multifaceted and potentially modality-dependent under complex audiovisual-task conditions.SIGNIFICANCE STATEMENT Pupil size and oscillatory α power are associated with cognitive demand and effort, but their relative sensitivity under complex audiovisual-task conditions is unclear, as is the extent to which they share underlying mechanisms. Using an audiovisual dual-task paradigm, we show that pupil size increases with increasing cognitive demands for both audition and vision. In contrast, changes in oscillatory α power depend on the respective task demands: parietal α power decreases with visual demand but not with auditory task demand. Hence, pupil size and α power show different sensitivity to cognitive demands, perhaps suggesting partly different underlying neural mechanisms.

Transformer for multiple object tracking: Exploring locality to vision

Multi-object tracking (MOT) is a critical task in various domains, such as traffic analysis, surveillance, and autonomous vehicles. The joint-detection-and-tracking paradigm has been extensively researched, which is faster and more convenient for training and deploying over the classic tracking-by-detection paradigm while achieving state-of-the-art performance. This paper explores the possibilities of enhancing the MOT system by leveraging the prevailing convolutional neural network (CNN) and a novel vision transformer technique Locality. There are several deficiencies in the transformer adopted for computer vision tasks. While the transformers are good at modeling global information for a long embedding, the locality mechanism, which learns the local features, is missing. This could lead to negligence of small objects, which may cause security issues. We combine the TransTrack MOT system with the locality mechanism inspired by LocalViT and find that the locality-enhanced system outperforms the baseline TransTrack by 5.3% MOTA on the MOT17 dataset.

Dynamic fry counting based on multi-object tracking and one-stage detection

Fry counting is an essential task in aquaculture that provides valuable information for feedstuff feeding, culture density adjustment, and fish economic benefit estimation. However, most current computer vision-based fry counting methods are designed for statically counting, with a fixed number of fry in a container. As a result, these methods are not suitable for dynamic scenarios, such as flowing water, where the effects of factors like water velocity and uneven lighting can cause errors in counting. To address this issue and improve accuracy, we propose a dynamic fry counting method capable of dealing with large-number fry counting. Specifically, we regard fry counting as a Multiple Object Tracking (MOT) problem based on Tracking-By-Detection (TBD) framework. Firstly, a dataset of streaming fry is constructed for training the detection model, and videos with varying numbers of fry are manually counted for counting tests. Secondly, due to the highly similar appearance of fry, Simple Online and Realtime Tracking (SORT) is combined with You Only Look Once version 5 Nano (YOLOv5-Nano) to perform fry tracking. To achieve stable tracking of fry movement, the SORT algorithm is improved based on multi-matching and trajectory recovery, reaching 82.6 % multi-object tracking accuracy. Finally, fry counting is realized based on the intersection of a counting line and moving trajectories. The results indicate that the proposed counting method achieves 96.4 % accuracy and can be implemented on CPU at 5–10 frames per second (fps) and GPU at 15–35 fps. In conclusion, the proposed method automatically counts the number of fry in videos with higher speed and accuracy, facilitating automated fishery farming.

Can Three-Dimensional Multiple Object Tracking Training Be Used to Improve Simulated Driving Performance? A Pilot Study in Young and Older Adults

Driving ability has been shown to be dependent on perceptual-cognitive abilities such as visual attention and speed of processing. There is mixed evidence suggesting that training these abilities may improve aspects of driving performance. This preliminary study investigated the feasibility of training three-dimensional multiple object tracking (3D-MOT)—a dynamic, speeded tracking task soliciting selective, sustained and divided attention as well as speed of processing—to improve measures of simulated driving performance in older and younger adults. A sample of 20 young adults (23–33 years old) and 14 older adults (65–76 years old) were randomly assigned to either a 3D-MOT training group or an active control group trained on a perceptual discrimination task as well as 2048. Participants were tested on a driving scenario with skill-testing events previously identified as optimal for cross-sectional comparisons of driving ability. Results replicated previously identified differences in driving behaviour between age groups. A possible trend was observed for the 3D-MOT trained group, especially younger adults, to increase the distance at which they applied their maximum amount of braking in response to dangerous events. This measure was associated with less extreme braking during events, implying that these drivers may have been making more controlled stops. Limitations of sample size and task realism notwithstanding, the present experiment offers preliminary evidence that 3D-MOT training might transfer to driving performance through quicker detection of or reaction to dangerous events and provides a rationale for replication with a larger sample size.

Wizard: Unsupervised goats tracking algorithm

Computer vision is an interesting tool for animal behavior monitoring, mainly because it limits animal handling and it can be used to record various traits using only one sensor. From previous studies, this technic has shown to be suitable for various species and behavior. However it remains challenging to collect individual information, i.e. not only to detect animals and behavior on the video frames, but also to identify them. Animal identification is a prerequisite to gather individual information in order to characterize individuals and compare them. A common solution to this problem, known as multiple objects tracking, consists in detecting the animals on each video frame, and then associate detections to a unique animal ID. Association of detections between two consecutive frames are generally made to maintain coherence of the detection locations and appearances. To extract appearance information, a common solution is to use a convolutional neural network (CNN), trained on a large dataset before running the tracking algorithm. For farmed animals, designing such network is challenging as far as large training dataset are still lacking. In this article, we proposed an innovative solution, where the CNN used to extract appearance information is parameterized using offline unsupervised training. The algorithm, named Wizard, was evaluated for the purpose of goats monitoring in outdoor conditions. 17 annotated videos were used, for a total of 4H30, with various number of animals on the video (from 3 to 8) and different level of color differences between animals. First, the ability of the algorithm to track the detected animals was evaluated. When animals were detected, the algorithm found the correct animal ID in 94.82% of the frames. When tracking and detection were evaluated together, we found that Wizard found the correct animal ID in 86.18% of the video length. In situations where the animal detection rate could be high, Wizard seems to be a suitable solution for individual behavior analysis experiments based on computer vision.

A Vision Detection Scheme Based on Deep Learning in a Waste Plastics Sorting System

The preliminary sorting of plastic products is a necessary step to improve the utilization of waste resources. To improve the quality and efficiency of sorting, a plastic detection scheme based on deep learning is proposed in this paper for a waste plastics sorting system based on vision detection. In this scheme, the YOLOX (You Only Look Once) object detection model and the DeepSORT (Deep Simple Online and Realtime Tracking) multiple object tracking algorithm are improved and combined to make them more suitable for plastic sorting. For plastic detection, multiple data augmentations are combined to improve the detection effect, while BN (Batch Normalization) layer fusion and mixed precision inference are adopted to accelerate the model. For plastic tracking, the improved YOLOX is used as a detector, and the tracking effect is further improved by optimizing the deep cosine metric learning and the metric in the matching stage. Based on this, virtual detection lines are set up to filter and extract information to determine the sorted objects. The experimental results show that the scheme proposed in this paper makes full use of vision information to achieve dynamic and real-time detection of plastics. The system is effective and versatile for sorting complex objects.

Study on the correlation between basketball players' multiple-object tracking ability and sports decision-making.

Players' multiple-object tracking (MOT) ability is very important in basketball because it may affect players' sports decision-making (SDM), thus affecting the results of the game. The purpose of this study was to investigate the differences between expert and novice basketball players in MOT ability and SDM and to explore the correlation between basketball players' visual attention and SDM. A total of 48 female basketball players (24 categorized in the expert group and 24 in the novice group) participated in the MOT task in Experiment 1 and the basketball 3 vs. 3 games in Experiment 2. Experiment 1 examined the difference in dynamic visual attention characteristics between expert players and novice players by changing the tracking number. Experiment 2 examined the differences between expert players and novice players through the SDM of basketball 3 vs. 3 games. Sports decisions were evaluated by basketball experts. MOT ability and SDM ability were analyzed through Pearson correlation. The overall MOT accuracy of expert players (64.6%) and novice players (55.7%) was significantly different (χ2 = 59.693, P = 0.000). There was no significant difference in accuracy when tracking 2-3 targets (P > 0.05), but there was a significant difference in accuracy when tracking 4-6 targets (P < 0.05). The overall SDM accuracy of expert players (91.6%) and novice players (84.5%) was significantly different (χ2 = 31.975, P = 0.000). There was no significant difference between expert players and novice players in the accuracy of dribbling decision-making (P > 0.05), but there was a significant difference in the accuracy of passing decision-making and shooting decision-making (P < 0.01). When tracking 4-5 targets, the tracking score was positively correlated with the passing decision score and dribbling decision score of expert players, and the tracking score of novice players was positively correlated with the passing decision score (r > 0.6, P < 0.01). First, the tracking accuracy of expert players was significantly higher than that of novice players, especially when tracking 4-6 targets. As the number of targets increased, accuracy decreased. Second, the accuracy of expert players' SDM was significantly higher than that of novice players, especially in passing decision-making and shooting decision-making. Expert players exhibited fast and accurate SDM. Third, there was a correlation between MOT ability and SDM performance. The MOT ability of 4-5 targets was positively correlated with passing decision-making, which was statistically significant. The correlation between the MOT ability and SDM performance of expert players was greater and more significant. Having too many targets to track (more than 6) interfered with players' decisions.

Pedestrian Detection and Tracking System Based on Deep-SORT, YOLOv5, and New Data Association Metrics

Pedestrian tracking and detection have become critical aspects of advanced driver assistance systems (ADASs), due to their academic and commercial potential. Their objective is to locate various pedestrians in videos and assign them unique identities. The data association task is problematic, particularly when dealing with inter-pedestrian occlusion. This occurs when multiple pedestrians cross paths or move too close together, making it difficult for the system to identify and track individual pedestrians. Inaccurate tracking can lead to false alarms, missed detections, and incorrect decisions. To overcome this challenge, our paper focuses on improving data association in our pedestrian detection system’s Deep-SORT tracking algorithm, which is solved as a linear optimization problem using a newly generated cost matrix. We introduce a set of new data association cost matrices that rely on metrics such as intersections, distances, and bounding boxes. To evaluate trackers in real time, we use YOLOv5 to identify pedestrians in images. We also perform experimental evaluations on the Multiple Object Tracking 17 (MOT17) challenge dataset. The proposed cost matrices demonstrate promising results, showing an improvement in most MOT performance metrics compared to the default intersection over union (IOU) data association cost matrix.

A Multi-Object Tracking Algorithm With Center-Based Feature Extraction and Occlusion Handling

For tracking suspicious objects using intelligent robots, Multiple Object Tracking (MOT) has gained great attention. MOT is easily affected by long-term severe occlusion. This work proposes a joint MOT algorithm to handle such occlusion. Pairs of frames in complicated environments are taken as input. A center-based feature extraction framework is designed for precisely detecting objects and extracting their feature maps. A ConvGRU module is applied to learn permanent representations by using historical spatio-temporal information of objects. A Hungarian matching method is applied to match the detected objects and predicted predictions. The proposed algorithm is compared with several representative methods on two public multi-object tracking benchmarks. Furthermore, this work constructs a database with videos captured from street scenarios and uses it to test the proposed algorithm and its peers. Experimental results demonstrate that the proposed algorithm outperforms its peers, especially under long-term severe occlusion, thus advancing the field of MOT.

Optimized Deep Learning for Congestion-Aware Continuous Target Tracking and Boundary Detection in IoT-Assisted WSN

Target tracking (TT) is an important application of Internet of Things (IoT)-assisted wireless sensor networks (WSNs) in which sensor nodes monitor and report the positions of continuously moving objects. However, continuous tracking of multiple objects causes network congestion. To solve this challenge and provide congestion-aware continuous TT and boundary detection, we propose a novel energy-aware radial clustering based piecewise regressive multiobjective golden eagle optimized deep convolutive learning (EARC-ODCL) algorithm. In particular, EARC-ODCL incorporates a multiobjective stochastic sampled golden eagle optimization to select the optimal cluster heads (CHs) for sending the target information. Also, a fully connected deep convolutive neural learning is used to identify the boundary of sensor nodes. Furthermore, a piecewise linear regression approach is used for network congestion prediction and minimize data loss. Thereby, the proposed EARC-ODCL is able to perform continuous TT and boundary detection with higher accuracy in IoT-assisted WSN. Extensive simulations are conducted to assess the performance of EARC-ODCL in terms of various metrics, such as TT accuracy and time, boundary detection accuracy (BDA), and data loss.

Multiple Object Tracking Using Re-Identification Model with Attention Module

Multi-object tracking (MOT) has gained significant attention in computer vision due to its wide range of applications. Specifically, detection-based trackers have shown high performance in MOT, but they tend to fail in occlusive scenarios such as the moment when objects overlap or separate. In this paper, we propose a triplet-based MOT network that integrates the tracking information and the visual features of the object. Using a triplet-based image feature, the network can differentiate similar-looking objects, reducing the number of identity switches over a long period. Furthermore, an attention-based re-identification model that focuses on the appearance of objects was introduced to extract the feature vectors from the images to effectively associate the objects. The extensive experimental results demonstrated that the proposed method outperforms existing methods on the ID switch metric and improves the detection performance of the tracking system.

EDDSN-MRT: multiple rodent tracking based on ear detection and dual siamese network for rodent social behavior analysis

BackgroundRodent social behavior is a commonly used preclinical model to interrogate the mechanisms underpinning various human neurological conditions. To investigate the interplay between neural systems and social behaviors, neuroscientists need a precise quantitative measure for multi-rodent tracking and behavior assessment in laboratory settings. However, identifying individual differences across multiple rodents due to visual occlusion precludes the generation of stable individual tracks across time.MethodsTo overcome the present limitations of multi-rodent tracking, we have developed an Ear Detection and Dual Siamese Network for Multiple Rodent Tracking (EDDSN-MRT). The aim of this study is to validate the EDDSN-MRT system in mice using a publicly available dataset and compare it with several current state-of-the-art methods for behavioral assessment. To demonstrate its application and effectiveness in the assessment of multi-rodent social behavior, we implemented an intermittent fasting intervention experiment on 4 groups of mice (each group is with different ages and fasting status and contains 8 individuals). We used the EDDSN-MRT system to track multiple mice simultaneously and for the identification and analysis of individual differences in rodent social behavior and compared our proposed method with Toxtrac and idtracker.ai.ResultsThe locomotion behavior of up to 4 mice can be tracked simultaneously using the EDDSN-MRT system. Unexpectedly, we found intermittent fasting led to a decrease in the spatial distribution of the mice, contrasting with previous findings. Furthermore, we show that the EDDSN-MRT system can be used to analyze the social behavior of multiple mice of different ages and fasting status and provide data on locomotion behavior across multiple mice simultaneously.ConclusionsCompared with several state-of-the-art methods, the EDDSN-MRT system provided better tracking performance according to Multiple Object Tracking Accuracy (MOTA) and ID Correct Rate (ICR). External experimental validation suggests that the EDDSN-MRT system has sensitivity to distinguish the behaviors of mice on different intermittent fasting regimens. The EDDSN-MRT system code is freely available here: https://github.com/fliessen/EDDSN-MRT.

Scheduling Framework for Accelerating Multiple Detection-Free Object Trackers

In detection-free tracking, after users freely designate the location of the object to be tracked in the first frame of the video sequence, the location of the object is continuously found in the following video frame sequence. Recently, technologies using a Siamese network and transformer based on DNN modules have been evaluated as very excellent in terms of tracking accuracy. The high computational complexity due to the usage of the DNN module is not a preferred feature in terms of execution speed, and when tracking two or more objects, a bottleneck effect occurs in the DNN accelerator such as the GPU, which inevitably results in a larger delay. To address this problem, we propose a tracker scheduling framework. First, the computation structures of representative trackers are analyzed, and the scheduling unit suitable for the execution characteristics of each tracker is derived. Based on this analysis, the decomposed workloads of trackers are multi-threaded under the control of the scheduling framework. CPU-side multi-threading leads the GPU to a work-conserving state while enabling parallel processing as much as possible even within a single GPU depending on the resource availability of the internal hardware. The proposed framework is a general-purpose system-level software solution that can be applied not only to GPUs but also to other hardware accelerators. As a result of confirmation through various experiments, when tracking two objects, the execution speed was improved by up to 55% while maintaining almost the same accuracy as the existing method.

Online multiple object tracking with enhanced Re‐identification

AbstractIn existing online multiple object tracking algorithms, schemes that combine object detection and re‐identification (ReID) tasks in a single model for simultaneous learning have drawn great attention due to their balanced speed and accuracy. However, different tasks require to focus different features. Learning two different tasks in the same model extracted features can lead to competition between the two tasks, making it difficult to achieve optimal performance. To reduce this competition, a task‐related attention network, which uses a self‐attention mechanism to allow each branch to learn on feature maps related to its task is proposed. Besides, a smooth gradient‐boosting loss function, which improves the quality of the extracted ReID features by gradually shifting the focus to the hard negative samples of each object during training is introduced. Extensive experiments on MOT16, MOT17, and MOT20 datasets demonstrate the effectiveness of the proposed method, which is also competitive in current mainstream algorithm.

Pedestrian multiple-object tracking based on FairMOT and circle loss

Multi-object Tracking is an important issue that has been widely investigated in computer vision. However, in practical applications, moving targets are often occluded due to complex changes in the background, which leads to frequent pedestrian ID switches in multi-object tracking. To solve the problem, we present a multi-object tracking algorithm based on FairMOT and Circle Loss. In this paper, HRNet is adopted as the baseline. Then, Polarized Self-Attention is added to HRNet-w32 to obtain weights of helpful information based on its modeling advantages. Moreover, the re-identification branch is optimized, and the Circle Loss is selected as the loss function to acquire more discriminative pedestrian features and to distinguish different pedestrians. The method proposed is assessed on the public MOT17 datasets. The experimental results show that the MOTA score achieves 69.5%, IDF1 reaches 70.0%, and the number of ID switches (IDs) decreases 636 times compared to the TraDes algorithm.

Hybrid Motion Model for Multiple Object Tracking in Mobile Devices

For an intelligent transportation system, multiple object tracking (MOT) is more challenging from the traditional static surveillance camera to mobile devices of the Internet of Things (IoT). To cope with this problem, previous works always rely on additional information from multivision, various sensors, or precalibration. Only based on a monocular camera, we propose a hybrid motion model to improve the tracking accuracy in mobile devices. First, the model evaluates camera motion hypotheses by measuring optical flow similarity and transition smoothness to perform robust camera trajectory estimation. Second, along the camera trajectory, smooth dynamic projection is used to map objects from image to world coordinate. Third, to deal with trajectory motion inconsistency, which is caused by occlusion and interaction of long time interval, tracklet motion is described by the multimode motion filter for adaptive modeling. Fourth, in tracklets association, we propose a spatiotemporal evaluation mechanism, which achieves higher discriminability in motion measurement. Experiments on MOT15, MOT17, and KITTI benchmarks show that our proposed method improves the trajectory accuracy, especially in mobile devices and our method achieves competitive results over other state-of-the-art methods.

Multiple object tracking based on quadratic graph matching

AbstractRecently, with the development of deep‐learning, the performance of multiple object tracking (MOT) algorithm based on deep neural networks has been greatly improved. However, it is still a difficult problem to successfully solve the tracking misalignment caused by occlusion and complex tracking scenes. Most of the work focusses on designing a sophisticated network, only little work focusses on data association. Actually, data association is very helpful in MOT. In this study, data association in tracking is taken as the main research task, and the authors introduce quadratic graph matching into MOT. Considering the objects in each frame as a graph, we can model the data association between two frames as a quadratic graph matching problem. And then it is transformed into a convex quadratic optimisation problem. Introducing high‐order structure features into the matching function effectively solves some tracking problems. The data association is designed into the tracking model as an overall solution. Experiments show that the proposed data association algorithm performs favourably against several state‐of‐the‐art data association methods and can be used in any tracking‐by‐detection method. Our code is publicly available from https://github.com/gjy0514/G_Model.

Disentangling working memory from multiple-object tracking: Evidence from dual-task interferences.

Multiple object tracking (MOT) is generally regarded as a pure attention-consuming task that draws heavily on attention resources. In the present study, we adopted a cross-channel visual-audio dual-task paradigm, i.e., the MOT task combined with the concurrent auditory N-back working memory task, to test whether working memory indeed plays a necessary role in the process of multiple tracking, as well as to further identify the specific types of working memory components involved in this process. Experiments 1a and 1b examined the relationship between the MOT task and nonspatial object working memory (OWM) processing by manipulatingthe tracking load and working memory load, respectively. Results in both experiments indicated that the concurrent nonspatial OWM task did not have a significant effect on the tracking capacity of the MOT task. In contrast, Experiments 2a and 2b examined the relationship between the MOT task and spatial working memory (SWM) processing by a similar approach. Results in both experiments indicated that the concurrent SWM task significantly impaired the tracking capacity of the MOT task, showing a gradual decrease with increasing SWM load. Overall, our study provides empirical evidence that multiple object tracking does involve working memory, primarily related to spatial working memory rather than nonspatial object working memory, which sheds more light on the mechanisms of multiple object tracking.