Multi-target Multi-camera Tracking Research Articles

Multi-Vehicle Multi-Camera Tracking With Graph-Based Tracklet Features

Multi-target multi-camera tracking (MTMCT) is an important application in intelligent transportation systems (ITS). The conventional works follow the tracking-by-detection scheme and use the information of the object image separately while matching the object from different cameras. As a result, the association information from the object image is lost. To utilize this information, we propose an efficient MTMCT application that builds features in the form of a graph and customizes graph similarity to match the vehicle objects from different cameras. We present algorithms for both the online scenario, where only the past images are used to match a vehicle object, and the offline scenario, where a given vehicle object is tracked with past and future images. For offline scenarios, our method achieves an IDF1-score of 0.8166 on the Cityflow dataset, which contains the actual scenes of the city from multiple street cameras. For online scenarios, our method achieves an IDF1-score of 0.75 with an FPS of 14. Our codes and datasets are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/elituan/GraphBasedTracklet_MTMCT</uri> .

Blockchain-Empowered Distributed Multicamera Multitarget Tracking in Edge Computing

Blockchain-Empowered Distributed Multicamera Multitarget Tracking in Edge Computing

Camera-aware re-identification feature for multi-target multi-camera tracking

Multi-target multi-camera tracking is an important research topic in intelligent surveillance to achieve re-identification (Re-ID) of moving targets across different cameras. However, Re-ID faces significant challenges owing to variations in cameras/viewpoints, making it difficult to learn discriminative feature representations for targets captured from different cameras/viewpoints. A new pipeline is introduced by balancing the across-cameras sample feature space in camera-aware Re-ID framework. Specifically, different sampling strategies play a crucial role on the performance under the same baseline and identify the feature discrepancy between cameras/viewpoints as a crucial factor. This proposed sampling strategy is called camera equalization sampling to learn enhanced feature disparities, which balances cameras under identity rather than randomly sampling in a batch. The sampled images are combined with non visual cues(cameras position encoding) to reduce intra class variance. For the mechanism of camera equalization sampling, the improved camera centric loss function can better reduce the negative impact of individual samples and provide stable learnable features.Our proposed method consists of three modules. (i) Camera equalization (CE) ensures that each batch collects at least one image from each camera for every identity, thereby enabling robust features. (ii) Camera position embedding (CPE) mitigates the scene bias caused by different cameras/viewpoints by encoding camera indices. (iii) Camera Center triplet loss(CCL) based on CE improves higher robustness to outliers and noisy labels. We demonstrated the proposed method's effectiveness on popular datasets such as DukeMTMC-reID, Market-1501, and MSMT17, achieving state-of-the-art performance.

Fast online multi-target multi-camera tracking for vehicles

Fast online multi-target multi-camera tracking for vehicles

Multicamera edge-computing system for persons indoor location and tracking

This paper presents an indoor person localization and tracking system that uses multiple smart cameras equipped with artificial intelligence (AI) accelerators serving as edge-computing nodes. Our main contributions are as follows: (a) the development of a new multicamera tracking system for indoor scenarios; (b) the release of a multitarget multicamera tracking dataset; and (c) the development of an annotation mechanism based on waypoints. The system can simultaneously track several individuals while preserving their privacy and anonymity, because no images or sensitive data are transmitted outside the edge nodes. Only the position and appearance of each person were transmitted to the central server. In addition, a multitarget multicamera tracking dataset was released. The dataset contains recordings from five cameras in an indoor scenario and is annotated with the real-world coordinates of individuals. Ground-truth annotations were semiautomatically generated using a mechanism in which people equipped with mobile phones followed specific paths with predefined waypoints. Software related to the ground-truth annotation mechanism has also been released as open source.

ALSA: Adaptive Low-light Correction and Self-Attention Module for Vehicle Re-Identification

Multi-Camera Vehicle Re-identification and Tracking (MCVRT) is a challenging task that involves identifying and tracking vehicles across multiple camera views in a surveillance network. Multi-Target Multi-Camera Tracking (MTMCT) and vehicle Re-Identification (Re-ID) are two major technologies applied to MCVRT tasks. Variations in aspect ratio, occlusion, orientation, and lighting conditions make vehicle re-identification and multi-camera tracking challenging. While some existing methods address these problems, it remains a significant challenge in the field. Additionally, most Re-ID datasets only include images captured in well-lit environments, and the impact of dark images on the performance of existing models needs to be explored further. This paper presents a new approach to address the challenge of low-light images in vehicle re-identification and achieves state-of-the-art results on public datasets. Our approach is based on two key components: (i) an Adaptive Low-light correction and Self-Attention module (ALSA) for image pre-processing in Vehicle Re-ID networks, and (ii) a new loss function called Log Triplet Loss (LT-Loss). We evaluated the presented approach through computer simulations on the VeRi-776 dataset, and the results showed that our model achieved a Rank@1 accuracy of 98.99%, and also outperformed commonly used models on dark images. Our study highlights the importance of considering lighting conditions in vehicle re-identification and provides a new approach to address this challenge.

Indoor Multipedestrian Multicamera Tracking Based on Fine Spatiotemporal Constraints

As indoor space is the primary place for pedestrian activities, obtaining intelligent monitoring of indoor pedestrians is crucial for intelligent video surveillance. Previous studies have verified the effectiveness of spatio-temporal constraints in multitarget multi-camera tracking (MTMCT). Pedestrians are generally subjected to fine spatio-temporal constraints within buildings, based on which the indoor geographic information system (GIS) technology can obtain automatic spatio-temporal modeling. Combined with artificial intelligence (AI) technology, we established a research framework of "GIS+AI+IMPMCT." Specifically, we proposed indoor multi-pedestrian multi-camera tracking (IMPMCT) based on fine spatio-temporal constraints. First, we used GIS to map the indoor monitoring images of buildings and automatically model the fine spatio-temporal relationship among the semantics of the entrance of the surveillance zone. Subsequently, we used the machine learning model of pedestrian localization and tracking to obtain local trajectories of pedestrians and combined them with map information to extract entrance semantics of trajectories. Finally, we used the local trajectory semantics and surveillance entrance semantic constraints to obtain a fine spatio-temporal constraint weight matrix between trajectories and fused pedestrian’s apparent features to obtain trajectory matching results. To verify our method, we established an IMPMCT dataset containing fine indoor spatio-temporal information. Our method obtained an IDF1 of 0.805, which is better than those of other methods. Furthermore, the tracking results obtained by the proposed method contained both image space and geospatial trajectories.

Real-Time Multi-Vehicle Multi-Camera Tracking with Graph-Based Tracklet Features

An essential application in intelligent transportation systems is multi-target multi-camera tracking (MTMCT), where the target’s activity is tracked from different cameras. Although the tracking-by-detection scheme is the primary paradigm in MTMCT, the object association information from the video frames is lost. This is mainly because the multi-camera multi-object matching uses the information from the video frames separately. To solve this problem and leverage this association information, we propose an MTMCT framework, where features are built in the form of a graph and a graph similarity algorithm is used to match multi-camera objects. In this paper, we focus on the real-time scenario, where only the past images are used to match an object. Our method achieves an IDF1 score (the ratio of the number of correctly identified objects to the number of ground truth and average objects) of 0.75 with a rate of 14 frames per second (fps).

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

Graph Neural Networks for Cross-Camera Data Association

Cross-camera image data association is essential for many multi-camera computer vision tasks, such as multi-camera pedestrian detection, multi-camera multi-target tracking, 3D pose estimation, etc. This association task is typically modeled as a bipartite graph matching problem and often solved by applying minimum-cost flow techniques, which may be computationally demanding for large data. Furthermore, cameras are usually treated by pairs, obtaining local solutions, rather than finding a global solution at once for all multiple cameras. Other key issue is that of the affinity function: the widespread usage of non-learnable pre-defined distances, such as the Euclidean and Cosine ones. This paper proposes an effective approach for cross-camera data-association focused on a global solution, instead of processing cameras by pairs. To avoid the usage of fixed distances and thresholds, we leverage the connectivity of Graph Neural Networks, previously unused in this scope, using a Message Passing Network to jointly learn features and similarity functions. We validate the proposal for pedestrian cross-camera association, showing results over the EPFL multi-camera pedestrian dataset. Our approach considerably outperforms the literature data association techniques, without requiring to be trained in the same scenario in which it is tested. Our code is available at <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://www-vpu.eps.uam.es/publications/gnn</uri></monospace>

An Improved End-to-End Multi-Target Tracking Method Based on Transformer Self-Attention

Current multi-target multi-camera tracking algorithms demand increased requirements for re-identification accuracy and tracking reliability. This study proposed an improved end-to-end multi-target tracking algorithm that adapts to multi-view multi-scale scenes based on the self-attentive mechanism of the transformer’s encoder–decoder structure. A multi-dimensional feature extraction backbone network was combined with a self-built raster semantic map which was stored in the encoder for correlation and generated target position encoding and multi-dimensional feature vectors. The decoder incorporated four methods: spatial clustering and semantic filtering of multi-view targets; dynamic matching of multi-dimensional features; space–time logic-based multi-target tracking, and space–time convergence network (STCN)-based parameter passing. Through the fusion of multiple decoding methods, multi-camera targets were tracked in three dimensions: temporal logic, spatial logic, and feature matching. For the MOT17 dataset, this study’s method significantly outperformed the current state-of-the-art method by 2.2% on the multiple object tracking accuracy (MOTA) metric. Furthermore, this study proposed a retrospective mechanism for the first time and adopted a reverse-order processing method to optimize the historical mislabeled targets for improving the identification F1-score (IDF1). For the self-built dataset OVIT-MOT01, the IDF1 improved from 0.948 to 0.967, and the multi-camera tracking accuracy (MCTA) improved from 0.878 to 0.909, which significantly improved the continuous tracking accuracy and reliability.

Lightweight Indoor Multi-Object Tracking in Overlapping FOV Multi-Camera Environments.

Multi-Target Multi-Camera Tracking (MTMCT), which aims to track multiple targets within a multi-camera network, has recently attracted considerable attention due to its wide range of applications. The main challenge of MTMCT is to match local tracklets (i.e., sub-trajectories) obtained by different cameras and to combine them into global trajectories across the multi-camera network. This paper addresses the cross-camera tracklet matching problem in scenarios with partially overlapping fields of view (FOVs), such as indoor multi-camera environments. We present a new lightweight matching method for the MTMC task that employs similarity analysis for location features. The proposed approach comprises two steps: (i) extracting the motion information of targets based on a ground projection method and (ii) matching the tracklets using similarity analysis based on the Dynamic Time Warping (DTW) algorithm. We use a Kanade–Lucas–Tomasi (KLT) algorithm-based frame-skipping method to reduce the computational overhead in object detection and to produce a smooth estimate of the target’s local tracklets. To improve matching accuracy, we also investigate three different location features to determine the most appropriate feature for similarity analysis. The effectiveness of the proposed method has been evaluated through real experiments, demonstrating its ability to accurately match local tracklets.

Soccer Player Video Target Tracking Based on Deep Learning

Soccer player video target tracking is a very challenging task, which has good practical and commercial value. Traditional soccer game target tracking relies on athletes to carry a recording chip to achieve target tracking, but the cost is very high. With the rapid development of photography technology and deep learning technology, athletes’ target tracking is realized through soccer game video. Deep learning technology is applied to computer vision detection and tracking. How to realize soccer players’ video target tracking under deep learning is a challenging lesson. To solve this problem, this paper takes the video target tracking of football players as the research object, collects the game images of the stadium through multiple cameras, realizes the long-term accurate tracking of multiple players, and establishes a multicamera multitarget tracking system. The KCF algorithm and the improved KCF algorithm formed by replacing the hog feature of the KCF algorithm with the depth convolution neural network are used to compare and analyze the impact of different target tracking ranges and target numbers on the target tracking accuracy of the system, so as to accurately obtain the motion trajectory of football players. The results show that the image data of football matches are collected independently by multiple cameras, and the data of multiple cameras are collected to generate each target motion datum. The KCF algorithm of multicamera multitarget tracking has good robustness and real-time for long-term accurate tracking of football players; the KCF algorithm and the improved KCF algorithm have high accuracy in target tracking. With the increase of tracking frame range, the accuracy of target tracking of the two algorithms is improved. At the same time, multitarget tracking helps to improve the antiocclusion ability of the system. The research results have important practical significance and good application prospects for the analysis technology of video content of football matches.

Online clustering-based multi-camera vehicle tracking in scenarios with overlapping FOVs

Multi-Target Multi-Camera (MTMC) vehicle tracking is an essential task of visual traffic monitoring, one of the main research fields of Intelligent Transportation Systems. Several offline approaches have been proposed to address this task; however, they are not compatible with real-world applications due to their high latency and post-processing requirements. This lack of suitable approaches motivates our proposal: A new low-latency online approach for MTMC tracking in scenarios with partially overlapping fields of view (FOVs), such as road intersections. Firstly, the proposed approach detects vehicles at each camera. Then, the detections are merged between cameras by applying cross-camera clustering based on appearance and location. Lastly, the clusters containing different detections of the same vehicle are temporally associated to compute the tracks on a frame-by-frame basis. The experiments show promising low-latency results while addressing real-world challenges such as the a priori unknown and time-varying number of targets and the continuous state estimation of them without performing any post-processing of the trajectories. Our code is available at http://www-vpu.eps.uam.es/publications/Online-MTMC-Tracking.

Adaptive Affinity for Associations in Multi-Target Multi-Camera Tracking.

Data associations in multi-target multi-camera tracking (MTMCT) usually estimate affinity directly from re-identification (re-ID) feature distances. However, we argue that it might not be the best choice given the difference in matching scopes between re-ID and MTMCT problems. Re-ID systems focus on global matching, which retrieves targets from all cameras and all times. In contrast, data association in tracking is a local matching problem, since its candidates only come from neighboring locations and time frames. In this paper, we design experiments to verify such misfit between global re-ID feature distances and local matching in tracking, and propose a simple yet effective approach to adapt affinity estimations to corresponding matching scopes in MTMCT. Instead of trying to deal with all appearance changes, we tailor the affinity metric to specialize in ones that might emerge during data associations. To this end, we introduce a new data sampling scheme with temporal windows originally used for data associations in tracking. Minimizing the mismatch, the adaptive affinity module brings significant improvements over global re-ID distance, and produces competitive performance on CityFlow and DukeMTMC datasets.

CLONE: Collaborative Learning on the Edges

The proliferation of edge computing technologies has boosted the development of new applications for a plethora of edge devices. However, many applications face privacy issues and bandwidth limitations. To solve these limitations, we propose a collaborative learning framework on the edges, named CLONE, which is steered by the real-world data sets collected from a large electric vehicle (EV) company and a grocery store of a shopping mall, respectively. We categorize two application scenarios for CLONE, i.e., CLONE in the training stage (CLONE_training) and CLONE in the inference stage (CLONE_inference). As to CLONE_training, we choose the failure prediction of EV battery and associated components as the first use case. While as for CLONE_inference, customer tracking in a grocery store is selected as another case study. In this work, the goal of the CLONE is to support real-time training and inference for connected vehicles and marketing intelligence services. Our experimental results on the EV data show that CLONE is able to reduce model training time without sacrificing algorithm performance. Furthermore, the experimental results on the video data from the grocery store reveal that CLONE is a useful approach to solve the multitarget multicamera tracking problem in a collaborative fashion.

Reinforcement Learning Based Querying in Camera Networks for Efficient Target Tracking

Surveillance camera networks are a useful monitoring infrastructure that can be used for various visual analytics applications, where high-level inferences and predictions could be made based on target tracking across the network. Most multi-camera tracking works focus on re-identification problems and trajectory association problems. However, as camera networks grow in size, the volume of data generated is humongous, and scalable processing of this data is imperative for deploying practical solutions. In this paper, we address the largely overlooked problem of scheduling cameras for processing by selecting one where the target is most likely to appear next. The inter-camera handover can then be performed on the selected cameras via re-identification or another target association technique. We model this scheduling problem using reinforcement learning and learn the camera selection policy using Q-learning. We do not assume the knowledge of the camera network topology but we observe that the resulting policy implicitly learns it. We evaluate our approach using NLPR MCT dataset, which is a real multi-camera multi-target tracking benchmark and show that the proposed policy substantially reduces the number of frames required to be processed at the cost of a small reduction in recall.

A novel multi-target multi-camera tracking approach based on feature grouping

Multi-target multi-camera tracking systems track many pedestrians through videos taken from multiple cameras. Generally, multi-target multi-camera tracking comprises three steps, namely, detection, feature extraction, and data association. It also involves a number of marginal post-processing procedures, such as pruning and interpolating. The task is a complicated and challenging problem. In this work, we mainly focus on the process of data association. When correlation clustering-based algorithms are adopted in data association, serious information loss may be observed, especially when pedestrians in a video run into occlusion. Thus, we propose a method called feature group which mitigates the decline in accuracy under occlusions. The proposed method is intuitional but easy to implement without changing the original framework. After comprehensive experiments, the proposed method is proved effective and is able to make substantial improvements on the DukeMTMC dataset. The feature group method is also competitive relative to other state-of-the-art methods.

Multi-Target Multi-Camera Tracking of Vehicles Using Metadata-Aided Re-ID and Trajectory-Based Camera Link Model.

In this paper, we propose a novel framework for multi-target multi-camera tracking (MTMCT) of vehicles based on metadata-aided re-identification (MA-ReID) and the trajectory-based camera link model (TCLM). Given a video sequence and the corresponding frame-by-frame vehicle detections, we first address the isolated tracklets issue from single camera tracking (SCT) by the proposed traffic-aware single-camera tracking (TSCT). Then, after automatically constructing the TCLM, we solve MTMCT by the MA-ReID. The TCLM is generated from camera topological configuration to obtain the spatial and temporal information to improve the performance of MTMCT by reducing the candidate search of ReID. We also use the temporal attention model to create more discriminative embeddings of trajectories from each camera to achieve robust distance measures for vehicle ReID. Moreover, we train a metadata classifier for MTMCT to obtain the metadata feature, which is concatenated with the temporal attention based embeddings. Finally, the TCLM and hierarchical clustering are jointly applied for global ID assignment. The proposed method is evaluated on the CityFlow dataset, achieving IDF1 76.77%, which outperforms the state-of-the-art MTMCT methods.

Multi-Target Multi-Camera Tracking With Optical-Based Pose Association

Multi-target multi-camera tracking (MTMCT) targets to generate trajectories of the object that appeared under multiple cameras automatically. MTMCT can be treated as a combination of intra-camera tracking and cross-camera tracking. The existing work only employs the global description to perform the tracklet generating. However, the global description cannot model the local similarity between targets, leading to existing methods not to be robust to occlusion and fast motion. To handle the mentioned problem, we propose an online Optical-based Pose Association (OPA) for multi-target multi-camera tracking. The proposed method utilizes local pose matching to solve the occlusion problem, and applies optical flow to reduce the distance caused by fast motion. For optical-based pose association, we firstly employ OpenPose to generate human pose for each proposal. Then, we utilize the optical flow generated by PWC-Net to adjust the estimated pose for the previous frame. Finally, the modified Object Keypoint Similarity is used to compute the similarity between the pose of the current frame and adjusted pose in the prior frame. Once obtaining the optical-based pose similarity, we combine it with the visual and bounding box spatial similarities to generate the final similarity matrix, and apply the Kuhn-Munkras algorithm for data association. The experiments on the MTMCT and MOT datasets verify the rationality of using human pose information and prove the superiority of the proposed method.