Multi-pedestrian Tracking Research Articles

A structural description of pedestrian movement behavior in multiple surveillance videos

This study purpose of recording pedestrians' activities in all surveillance videos and then convert these records into structured descriptions to analyze a large number of surveillance videos. This paper proposes a four-stage processing method. The first stage uses convolutional neural network (CNN) to detect objects in the surveillance video and classify them. The second stage finds the correlation of pedestrians in continuous images and describes multiple objects' trajectories in the surveillance video. The third stage calculates the similarity of the pedestrian object sets of the surveillance video at different times as the basis for pedestrian re-identification. In the fourth stage, extract the walking path of the pedestrian in each surveillance video. Each pedestrian's actions in each surveillance video, including whether to stand, walking path, direction, clothing color and walking time. To construct a correlation table of pedestrian motion in multiple surveillance videos.This experiment uses two image datasets for experimentation and analysis to evaluate this method's effectiveness for pedestrian movement in multiple surveillance videos. The image datasets are the Multi-Camera Object Tracking (MCT) Challenge and PETS 2009 Benchmark Data. This experiment includes pedestrian detection, pedestrian identification, and multi-pedestrian tracking of pedestrians in single and multiple surveillance cameras. The results show that in the pedestrian detection experiment, the convolutional neural network can effectively solve the problems caused by static objects, light changes, similar colors, crowded pedestrians, and incomplete objects. In the pedestrian recognition and re-recognition experiments, the accuracy is reduced due to the influence of low-resolution surveillance videos. This detection method of the pedestrian is more accurate than traditional dynamic object detection methods. After the pedestrian tracking and re-identification are completed, the pedestrian relationships in multiple surveillance cameras are integrated, and a pedestrian behavior association table is built for surveillance videos.

Rlm-tracking: online multi-pedestrian tracking supported by relative location mapping

The challenge of multi-object tracking stands as a fundamental focus in computer vision research, finding widespread applications in areas such as public safety, transportation, autonomous vehicles, robotics, and other domains involving artificial intelligence. Given the intricate nature of natural scenes, the occurrence of object occlusion and semi-occlusion is commonplace in basic tracking tasks. These factors often result in challenges such as ID switching, object loss, detection errors, and misaligned bounding boxes, thereby significantly impacting the precision of multi-object tracking.This paper aims to address the aforementioned issues and proposes a novel multi-object tracker, incorporating Relative location mapping (RLM) and Target region density (TRD) modeling. The new tracker is more sensitive to differences in the spatial relationships between targets, allowing it to dynamically introduce low-scoring detection boxes into different regions based on the density of target regions in the image. This improves the accuracy of target tracking while avoiding the consumption of a significant amount of computational resources.Our research results indicate that when applying this method to state-of-the-art multi-object tracking approaches, the proposed model achieves improvements of 0.4 to 0.8 points in the HOTA and IDF1 metrics on the MOT17 and MOT20 datasets. This demonstrates the effectiveness of the proposed method in enhancing multi-object tracking performance.

Multi-pedestrian Tracking Method Fusing Two-stage Matching

Multi-pedestrian tracking is one of the hot topics in computer vision. For an intelligent mobile robot, multi-pedestrian tracking from a first-person perspective can provide information for navigating through a crowd and ensure safety. Most of the existing methods cannot deal with occlusion and trajectory overlap well. In this paper, a multi-pedestrian tracking method fusing two-stage matching is proposed. Firstly, the detection and the corresponding feature values of the pedestrians are obtained by a multi-task learning network based on CenterNet. Then the detected pedestrians are matched with feature values by greedy strategy. When dealing with the reappearance of pedestrians caused by occlusion or trajectory overlap, the sample database is established to update the samples in real time. The color histogram and HOG feature are calculated for each sample. When the pedestrian disappears, the direction of disappearance and the last position is recorded for the selection of trajectory. Finally, the KM algorithm is used for cross-frame matching. Our method is compared with some recent methods on MOT data sets. The result shows that our method has a significant improvement in the main evaluation index MOTA.

IDT: An Integration of Detection and Tracking Toward Low-Observable Multipedestrian for Urban Autonomous Driving

Robust pedestrian trajectory-tracking is an essential prerequisite to traffic accident prevention. However, it is a challenging task in urban autonomous driving, since the weak backscattered signals from pedestrians with small radar cross-section may be submerged in strong background clutters, especially under adverse weather conditions. On this account, this article presents an integration of detection and tracking (iDT) toward multipedestrian with a low signal-to-noise ratio (SNR). In particular, in contrast to conventional methods, in which the detection and tracking are treated as two separate processes, we address them jointly to ensure the accuracy of continuous detection and tracking in low SNR conditions. Another distinguishing element is that to accommodate the time-varying number of targets, the Bayesian framework is tailored by augmenting the state vector with a multipedestrian evolutional indicator. The advantage is that all targets can be tracked simultaneously by searching the global likelihood ratio of a spectrum once, rather than assigning an individual tracker to each target in conventional methods. Furthermore, through the proposed integrated framework, the data association problem is circumvented because there is no explicit measurement-target assignment process in our approach. In addition, a commercial automotive multiple-input-multiple-output millimeter-wave radar sensor is employed to validate the proposed method. Consequently, numerous simulation and experiment results turn out that iDT shows unique advantages in low-observable multipedestrian tracking compared with traditional methods.

Online Multipedestrian Tracking Based on Fused Detections of Millimeter Wave Radar and Vision

This paper focuses on the challenge of pedestrian tracking using camera and millimeter wave radar in autonomous driving. Pedestrian tracking using a single sensor has inherent limitations due to the lack of comprehensive dimensionality of tracking information. Meanwhile, existing multi-sensor based tracking algorithms suffer from limited tracking accuracy by applying the fusion of projected positions. To enhance the tracking accuracy and robustness, a multi-sensor based tracking algorithm based on fused detection of millimeter-wave radar and vision is proposed, which improves the association of detection results from multiple heterogeneous sensors by utilizing newly designed back-projection and undirected graph, and finally improves the fusion detection by simultaneously utilizing a pedestrian’s appearance, local and global location information. Field tests are conducted to produce dataset, and the performance evaluation results based on the self-produced dataset have verified the superiority of the proposed algorithm over the conventional single-sensor based tracking algorithm and multi-sensor based tracking algorithms.

Contact-Free Multitarget Tracking Using Distributed Massive MIMO-OFDM Communication System: Prototype and Analysis

Wireless-based human activity recognition has become an essential technology that enables contact-free human-machine and human-environment interactions. In this paper, we consider contact-free multi-target tracking (MTT) based on available communication systems. A radar-like prototype is built upon a sub-6 GHz distributed massive multiple-input and multiple-output (MIMO) orthogonal frequency-division multiplexing (OFDM) communication system. Specifically, the raw channel state information (CSI) is calibrated in the frequency and antenna domain before being used for tracking. Then the targeted CSIs reflected or scattered from the moving pedestrians are extracted. To evade the complex association problem of distributed massive MIMO-based MTT, we propose to use a complex Bayesian compressive sensing (CBCS) algorithm to estimate the targets’ locations based on the extracted target-of-interest CSI signal directly. The estimated locations from CBCS are fed to a Gaussian mixture probability hypothesis density (GM-PHD) filter for tracking. A multi-pedestrian tracking experiment is conducted in a room with a size of 6.5 m×10 m to evaluate the performance of the proposed algorithm. According to experimental results, we achieve 75th and 95th percentile accuracy of 12.7 cm and 18.2 cm for single-person tracking and 28.9 cm and 45.7 cm for multi-person tracking, respectively. Furthermore, the proposed algorithm achieves tracking purposes in real-time, which is promising for practical MTT use cases.

Improving Multiple Pedestrian Tracking in Crowded Scenes with Hierarchical Association

Recently, advances in detection and re-identification techniques have significantly boosted tracking-by-detection-based multi-pedestrian tracking (MPT) methods and made MPT a great success in most easy scenes. Several very recent works point out that the two-step scheme of first detection and then tracking is problematic and propose using the bounding box regression head of an object detector to realize data association. In this tracking-by-regression paradigm, the regressor directly predicts each pedestrian's location in the current frame according to its previous position. However, when the scene is crowded and pedestrians are close to each other, the small and partially occluded targets are easily missed. In this paper, we follow this pattern and design a hierarchical association strategy to obtain better performance in crowded scenes. To be specific, at the first association, the regressor is used to estimate the positions of obvious pedestrians. At the second association, we employ a history-aware mask to filter out the already occupied regions implicitly and look carefully at the remaining regions to find out the ignored pedestrians during the first association. We integrate the hierarchical association in a learning framework and directly infer the occluded and small pedestrians in an end-to-end way. We conduct extensive pedestrian tracking experiments on three public pedestrian tracking benchmarks from less crowded to crowded scenes, demonstrating the proposed strategy's effectiveness in crowded scenes.

A Multi-Pedestrian Tracking Algorithm for Dense Scenes Based on an Attention Mechanism and Dual Data Association

Aiming at the problems of frequent identity switches (IDs) and trajectory interruption of multi-pedestrian tracking algorithms in dense scenes, this paper proposes a multi-pedestrian tracking algorithm based on an attention mechanism and dual data association. First, the FairMOT algorithm is used as a baseline to introduce the feature pyramid network in the CenterNet detection network and up-sampling the output multi-scale fused feature maps, effectively reducing the rate of missed detection of small-sized and obscured pedestrians. The improved channel attention mechanism module is embedded in the CenterNet’s backbone network to improve detection accuracy. Then, a re-identification (ReID) branch is embedded in the head of the detection network, and the two sub-tasks of pedestrian detection and pedestrian apparent feature extraction are combined in a multi-task joint learning approach to output the pedestrian apparent feature vectors while detecting pedestrians, which improves the computational efficiency and localization accuracy of the algorithm. Finally, we propose a dual data association tracking model that tracks by associating almost every detection box instead of only the high-scoring ones. For low-scoring detection boxes, we utilize their similarities with trajectories to recover obscured pedestrians. The experiment using the MOT17 dataset shows that the tracking accuracy is improved by 0.6% compared with the baseline FairMOT algorithm, and the number of switches decreases from 3303 to 2056, which indicates that the proposed algorithm can effectively reduce the number of trajectory interruptions and identity switching.

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

Accurate and reliable tracking of multi-pedestrian is of great importance for autonomous driving, human-robot interaction and video surveillance. Since different scenarios have different best-performing sensors, sensor fusion perception plans are believed to have complementary modalities and be capable of handling situations which are challenging for single sensor. In this paper, we propose a novel track-to-track fusion strategy for multi-pedestrian tracking by using a millimeter-wave (MMW) radar and a monocular camera. Pedestrians are firstly tracked by each sensor according to the sensor characteristic. Specifically, the 3D monocular pedestrian detections are obtained by a convolutional neural network (CNN). The trajectory is formed by the tracking-by-detection approach, combined with Bayesian estimation. The measurement noise of the 3D monocular detection is modeled by a detection uncertainty value obtained from the same CNN, as an approach to estimate the pedestrian state more accurately. The MMW radar utilizes the track-before-detection method due to the sparseness of the radar features. Afterwards, the pedestrian trajectories are obtained by the proposed track-to-track fusion strategy, which can work adaptively under challenging weather conditions, low-illumination conditions and clutter scenarios. A group of tests are carried out to validate our pedestrian tracking strategy. Tracking trajectories and optimal sub-pattern assignment (OSPA) metric demonstrate the accuracy and robustness of the proposed multi-sensor multi-pedestrian tracking system.

Multi-Pedestrian Tracking in Crowded Scenes by Modeling Movement Behavior and Optimizing Kalman Filter

Most multi-object tracking methods have achieved good results in tracking multiple pedestrians with Kalman filter, but their tracking performance in crowded scenes is still poor due to pedestrian avoidance and frequent occlusion. In crowded scenes, the pedestrian trajectory prediction with Kalman filter alone is unreliable. In this paper, a two-dimensional field-of-view avoidance force model (AFM) is proposed to assist the Kalman filter prediction by sensing the avoidance force and then complete pedestrian tracking. In the model, each pedestrian has a two-dimensional field of view to perceive the avoidance force, which determines the next predicted trajectory. In real scenes, pedestrians tend to be more concerned about the surrounding area, so different areas are set to simulate the attention mechanisms of pedestrians in real scenes. In the FairMOT model, AFM is used to optimize the pedestrian state values of Kalman filter prediction and the optimized model is trained on the MOT16 public dataset. The experimental results on the MOT20 benchmark dataset show that compared with the mainstream tracking model FairMOT, our method respectively improves MOTA by 2.7% and IDF1 by 2.2%. Our method also achieves the good performance on MOT15, MOT16, and MOT17 tracking benchmarks.

A Multi-Pedestrian Tracking Algorithm Based on Center Point Detection and Person Re-identification

Objectives In video-based multiple object tracking, the object detection and re-identification have a strong correlation. The existing methods generally train the object detection and re-identification networks separately, which makes the tracking speed fail to meet the real-time requirements. In this paper, we integrate the detection and re-identification into one network to accelerate the tracking process, and also solve the problems of identity switching and failure tracking. Methods This paper develops a pedestrian motion model and obtains the optimal state estimation of pedestrians using center point detection. The person re-identification model with deep layer features uses the Mahalanobis distance and cosine distance to enhance the ability of person identification. And the Hungary algorithm is used for data online association, where the state estimation results become more accurate using Kalman filtering, and the unrelated lost objects are predicted by motion. Results Experiments are conducted on MOT15, MOT16 and MOT17 datasets using the proposed algorithm and other multi-pedestrian tracking algorithms, and the multiple object tracking accuracy of tracking results using our proposed algorithm is 63.5, 72.4 and 70.9, respectively, and the identity F1-measure is optimal, with the real-time rate. Conclusions The proposed algorithm can accelerate the tracking speed by network parameter sharing, and improve the recognition accuracy by person re-identification training.

A novel social distancing analysis in urban public space: A new online spatio-temporal trajectory approach

Social distancing in public spaces plays a crucial role in controlling or slowing down the spread of coronavirus during the COVID-19 pandemic. Visual Social Distancing (VSD) offers an opportunity for real-time measuring and analysing the physical distance between pedestrians using surveillance videos in public spaces. It potentially provides new evidence for implementing effective prevention measures of the pandemic. The existing VSD methods developed in the literature are primarily based on frame-by-frame pedestrian detection, addressing the VSD problem from a static and local perspective. In this paper, we propose a new online multi-pedestrian tracking approach for spatio-temporal trajectory and its application to multi-scale social distancing measuring and analysis. Firstly, an online multi-pedestrian tracking method is proposed to obtain the trajectories of pedestrians in public spaces, based on hierarchical data association. Then, a new VSD method based on spatio-temporal trajectories is proposed. The proposed method not only considers the Euclidean distance between tracking objects frame-by-frame but also takes into account the discrete Fréchet distance between trajectories, hence forms a comprehensive solution from both static and dynamic, local and holistic perspectives. We evaluated the performance of the proposed tracking method using the public dataset MOT16 benchmark. We also collected our own pedestrian dataset “SCU-VSD” and designed a multi-scale VSD analysis scheme for benchmarking the performance of the social distancing monitoring in the crowd. Experiments have demonstrated that the proposed method achieved outstanding performance on the analysis of social distancing.

Near-Online Multi-Pedestrian Tracking via Combining Multiple Consistent Appearance Cues

An important cue for multi-pedestrian tracking in video is the consistent appearance of an individual for quite a while. In this paper, we address multi-pedestrian tracking by learning a robust appearance model from the paradigm of tracking by detection. To separate detections of different pedestrians while assembling detections of the same pedestrian, we take advantage of the cue of consistent appearance and exploit three types of evidence from the recent, past and near-future. Existing online approaches only exploit the detection-to-detection and sequence-to-detection metrics, which focus on the recent and past appearance patterns respectively, while the future pedestrian appearance is simply ignored. This drawback is remedied in this paper by further considering the sequence-to-sequence metric, which resorts to near-future appearance presentation. Adaptive combination weights are learned to fuse these three different metrics. Moreover, we propose a novel Focal Triplet Loss to make the model focus more on hard examples than the easy ones. We demonstrate that this can significantly enhance the discriminating power of the model compared with treating every sample equally. Effectiveness and efficiency of the proposed method is verified by conducting comprehensive ablation studies and comparing with many competitive (offline/online/near-online) counterparts on the MOT16 and MOT17 Challenges.

Regaining Sight of Humanity on The Roadway towards Automation

In this study, we propose a new multi-pedestrian tracking (MPT) method that performs quickly and efficiently track pedestrians in real-time system. The proposed method considers combining shallow convolutional neural networks (CNN) with ensemble learning method, Siamese random forests (SRF). Unlike conventional methods, to promote robustness of ensemble method, feature transformation is applied which exploit shallow networks in appearances of still images to extract enrich features. We formulate the problem of MOT in a structured learning framework based on SRF. Each forest learns differences of random feature pairs, which are extracted from the former process to enhance robustness to easily happened circumstances in a moving vehicle. When it compares to the conventional tracking algorithms, the proposed approach, based on SRF, takes advantage of lightweight and efficiency. The proposed lightweight multiple pedestrian tracker was successfully applied to benchmark datasets and yielded a similar or better performance level as compared with state-of-theart methods.

Multi-pedestrian tracking optimized by social force model under first-person perspective

Multi-pedestrian tracking optimized by social force model under first-person perspective

An Efficient Edge Artificial Intelligence MultiPedestrian Tracking Method With Rank Constraint

Characterized by the ability to handle varying number of objects, tracking by detection framework becomes increasingly popular in multiobject tracking (MOT) problem. However, the tracking performance heavily depends on the object detector. Considering that data association optimization and association affinity model are two key parts in MOT, an online multipedestrian tracking method is proposed to formulate a more effective association affinity model. It includes a two-step data association taking advantage of rank-based dynamic motion affinity model. The rank-based dynamic motion affinity model is used to estimate the object state and refine the trajectory for each of target to achieve the noiseless trajectory. Both strategies are beneficial to eliminate ambiguous detection responses during association. To fairly verify the proposed method, three public datasets are adopted. Both qualitative and quantitative experiment results demonstrate the superiorities of the proposed tracking algorithm in comparison with its counterparts.

Infrared Multi-Pedestrian Tracking in Vertical View via Siamese Convolution Network

Target tracking has become one of the research hotspots in the field of computer vision in recent years. In this paper, a new intelligent algorithm of infrared multi-pedestrian tracking in vertical view is proposed. In the algorithm, the pedestrians in the infrared image can be quickly detected and located with the method of the Faster Regions with CNN features (RCNN) and then are tracked with the improved Siamese network. The tracking method based on Siamese network transforms the tracking problem into a similarity verification problem and evaluates the similarity score between new frame feature and target frame feature by convolution network. The candidate region with the highest score is considered as the current position of the target. In this paper, the Siamese network is combined with Faster RCNN for multi-pedestrian tracking. In addition, the tracking results of adjacent frames are introduced into the similarity evaluation of current frames to improve the tracking accuracy when the pedestrian posture changes. The experimental results show that the algorithm has good robustness and tracking result and achieves competitive performance.

Multi-Pedestrian Tracking Based on Improved Two Step Data Association

The multi-object tracking (MOT) algorithms based on tracking by detection framework are the state-of-the-art trackers in recent years. Association optimization and association affinity model are two key parts in MOT, which have attracted attention to build effective association model to overcome ambiguous detection responses. In this paper, we have proposed an online multi-pedestrian tracking algorithm that uses a two-step data association with the help of improved sparse based appearance affinity model and rank based motion affinity model. The association framework is constructed by fusing the trajectory dynamic information and confidence based two-step data associations. The missing frames of a tracklet are counted based on the sparse reconstruction error of a target. An incremental SVD and downdate SVD decomposition is devised to estimate the rank of the Hankel matrix in rank based motion model. The estimated result is fed back to compute the tracklets confidence during association optimization. Both those strategies are beneficial to eliminate ambiguous detection responses during association. By this association optimization strategy, the fragmented tracklets in online tracking are reduced in some extent. We evaluate our method on four public available challenging datasets. The experimental results, both qualitative and quantitative, demonstrate that the proposed tracking algorithm has a good tracking performance compared with several state-of-the-art multi-object trackers.

Fast Online Multi-Pedestrian Tracking via Integrating Motion Model and Deep Appearance Model

In recent years, multi-object tracking has attracted more and more attention, both in academia and engineering, but most of the recent works do not pay attention to the speed of the algorithm and only pursue the accuracy. In this paper, we propose an online multi-pedestrian tracking algorithm, taking into account both the accuracy and the speed. First, the motion models of the targets are established by the Kalman filter. At the same time, the appearance models of the targets are extracted by the convolutional neural network. Moreover, a data association algorithm is proposed, which integrates the motion information, including scale, intersection-over-union, and distance, and the appearance information, including the current appearance model and the long-term appearance model. With the data association algorithm, the matching between detections and tracklets is realized, and the goal of tracking by detection is achieved. We compare the proposed algorithm with other algorithms on the MOT15 benchmark and the MOT16 benchmark. The experiment results show that the algorithm has high accuracy and good real-time performance.

Real-Time Pedestrian Tracking and Counting with TLD

This paper describes a solution to solve the issue of automatic multipedestrian tracking and counting. First, background modeling algorithm is applied to actively obtain multipedestrian candidates, followed by a confirmation step with classification. Then each pedestrian patch is handled by real-time TLD (Tracking-Learning-Detection) to get a new predication position according to similarity measure. Further TLD results are compared with classification list to determine a new, disappeared, or existing pedestrian. Finally single line counting with buffer zone is employed to count pedestrians. Experiments results on the public database, PETS, demonstrate the validity of our solution.