Multi-camera Tracking Research Articles

Trusted outdoor multi-camera tracking system powered by FPGA

Computer vision has become a crucial area of research due to the increasing use of electronics, information technology, and network communication. Adding more algorithms to the process of locating text, faces, vehicles, and other moving objects can improve the effectiveness of the search when compared to a "ground truth" reference. This work aims to track a moving object as it passes through the fields of view of multiple outdoor cameras. Both the temporal difference algorithm and the fixed background algorithm have been used in the search process for objects in a video with a frame dimension of 120 by 160 pixels. The FPGA panel used in the system was the Xilinx Spartan-6 LX45T. This method can speed up the overall process because it does not require the objects to be registered first. The proposed approach is highly resistant to object orientation and robust, with an error rate of less than (0.05), resulting in the best possible results in terms of global recognition. Consequently, it attains exceptional global recognition results while maintaining its reliable performance.

LaMMOn: language model combined graph neural network for multi-target multi-camera tracking in online scenarios

Multi-target multi-camera tracking is crucial to intelligent transportation systems. Numerous recent studies have been undertaken to address this issue. Nevertheless, using the approaches in real-world situations is challenging due to the scarcity of publicly available data and the laborious process of manually annotating the new dataset and creating a tailored rule-based matching system for each camera scenario. To address this issue, we present a novel solution termed LaMMOn, an end-to-end transformer and graph neural network-based multi-camera tracking model. LaMMOn consists of three main modules: (1) Language Model Detection (LMD) for object detection; (2) Language and Graph Model Association module (LGMA) for object tracking and trajectory clustering; (3) Text-to-embedding module (T2E) that overcome the problem of data limitation by synthesizing the object embedding from defined texts. LaMMOn can be run online in real-time scenarios and achieve a competitive result on many datasets, e.g., CityFlow (HOTA 76.46%), I24 (HOTA 25.7%), and TrackCUIP (HOTA 80.94%) with an acceptable FPS (from 12.20 to 13.37) for an online application.

Turning the operating room into a mixed-reality environment: a prospective clinical investigation for cerebral aneurysm clipping.

The overall aim of this study was to demonstrate the potential benefit of a novel mixed-reality-head-mounted display (MR-HMD) on the spatial orientation of surgeons. In a prospective clinical investigation, the authors applied for the first time a new multicamera navigation technology in an operating room setting that allowed them to directly compare MR-HMD navigation to standard monitor navigation. In the study, which included 14 patients with nonruptured middle cerebral artery aneurysms, the authors investigated how intuitively and effectively surgical instruments could be guided in 5 different visual navigation conditions. The authors demonstrate that multicamera tracking can be reliably integrated in a clinical setting (usability score 1.12 ± 0.31). Moreover, the technology captures large volumes of the operating room, allowing the team to track and integrate different devices and instruments, including MR-HMDs. Directly comparing mixed-reality navigation to standard monitor navigation revealed a significantly improved intuition in mixed reality, leading to navigation times that were twice as fast (2.1×, p ≤ 0.01). Despite the enhanced speed, the same targeting accuracy (approximately 2.5 mm, freehand tool use) in comparison to monitor navigation could be observed. Intraoperative planning strategies with mixed reality clearly outperformed classic preoperative planning: surgeons scored the mixed-reality plan as the best trajectory in 63% of the cases (chance level 33%). The incorporation of mixed reality in neurosurgical operations marks a significant advancement in the field. The use of mixed reality in brain surgery enhances the spatial awareness of surgeons, enabling more instinctive and precise surgical interventions. This technological integration promises to refine the execution of complex procedures without compromising accuracy.

Proposing Smart System for Detecting and Monitoring Vehicle Using Multiobject Multicamera Tracking

Nowadays, the strong development of the economy and society has driven the increase in traffic participation, making traffic management increasingly difficult. To effectively address this issue, AI applications are being applied to improve urban traffic management and operations. Therefore, we propose a smart system to detect and monitor vehicles across multiple surveillance cameras. Our system leverages data collected from traffic surveillance cameras and harnesses the power of deep learning technology to detect and track vehicles smoothly. To achieve this, we use the YOLO model for detection in conjunction with the DeepSORT algorithm for precise vehicle tracking on each camera. Furthermore, our system uses a ResNet backbone model for feature extraction of objects within each camera’s frame. It utilizes cosine distance to identify similar objects in other cameras, facilitating multicamera tracking. To ensure optimal performance, our system is implemented using the NVIDIA DeepStream SDK, enabling it to achieve an impressive speed of 21 fps on each camera and an average of precision approximately 85% for three modules. The results of our study affirm the system’s suitability and its potential for practical applications in the field of urban traffic management.

High-Precision Visual Servoing for the Neutron Diffractometer STRESS-SPEC at MLZ.

With neutron diffraction, the local stress and texture of metallic components can be analyzed non-destructively. For both, highly accurate positioning of the sample is essential, requiring the measurement at the same sample location from different directions. Current sample-positioning systems in neutron diffraction instruments combine XYZ tables and Eulerian cradles to enable the accurate six-degree-of-freedom (6DoF) handling of samples. However, these systems are not flexible enough. The choice of the rotation center and their range of motion are limited. Industrial six-axis robots have the necessary flexibility, but they lack the required absolute accuracy. This paper proposes a visual servoing system consisting of an industrial six-axis robot enhanced with a high-precision multi-camera tracking system. Its goal is to achieve an absolute positioning accuracy of better than 50μm. A digital twin integrates various data sources from the instrument and the sample in order to enable a fully automatic measurement procedure. This system is also highly relevant for other kinds of processes that require the accurate and flexible handling of objects and tools, e.g., robotic surgery or industrial printing on 3D surfaces.

GaitFormer: Leveraging dual-stream spatial–temporal Vision Transformer via a single low-cost RGB camera for clinical gait analysis

Gait analysis is an essential technique in treating patients with lower limb dysfunctions. Traditional methods often rely on expensive and complex equipment, such as wearable body sensors and a multi-camera with marker tracking system. Aiming for a more cost-effective yet accurate alternative, this paper introduces GaitFormer, a novel approach that leverages Vision Transformer (ViT) for gait analysis using minimal, non-invasive equipment, i.e. a single low-cost RGB camera. Initially, a unique dataset using a multi-camera system with marker tracking, comprising 6 walking patterns gathered from 80 volunteers is developed. The ViT-based GaitFormer is then proposed to automatically recognize human walking patterns through a single RGB camera. GaitFormer comprises hybrid networks for each step, including: (i) a cascaded convolutional 2D human key points estimation network; (ii) a ViT-based dual-stream spatial–temporal network extending the information of human key points into 3D; (iii) leveraging specific lower limb key joints’ angle features for clinical gait analysis, capturing the geometric, kinematic, and physical attributes of human motion; (iv) employing a pure self-attention-based classification network to recognize clinical human walking patterns. The experiments are designed to comprehensively validate each step against various related baseline methods and multi-camera tracking system, with results demonstrating the promising performance of GaitFormer as an affordable, precise, and integrated solution. To the best of our knowledge, GaitFormer is the first hybrid CNN- and ViT-based end-to-end solution via low-cost device for clinically valuable gait analysis.

Multi-camera tracking of mechanically thrown objects for automated in-plant logistics by cognitive robots in Industry 4.0

AbstractEmploying cognitive robots, capable of throwing and catching, is a strategy aimed at expediting the logistics process within Industry 4.0’s smart manufacturing plants, specifically for the transportation of small-sized manufacturing parts. Since the flight of mechanically thrown objects is inherently unpredictable, it is crucial for the catching robot to observe the initial trajectory with utmost precision and intelligently forecast the final catching position to ensure accurate real-time grasping. This study utilizes multi-camera tracking to monitor mechanically thrown objects. It involves the creation of a 3D simulation that facilitates controlled mechanical throwing of objects within the internal logistics environment of Industry 4.0. The developed simulation empowers users to define the attributes of the thrown object and capture its trajectory using a simulated pinhole camera, which can be positioned at any desired location and orientation within the in-plant logistics environment of flexible manufacturing systems. The simulation facilitated ample experimentation to be conducted for determining the optimal camera positions for accurately observing the 3D interception positions of a flying object based on its apparent size on the camera’s sensor plane. Subsequently, a variety of calibrated multi-camera setups were experimented while placing cameras at identified optimal positions. Based on the obtained results, the most effective multi-camera configuration setup is derived. Finally, a training dataset is prepared for 3000 simulated throwing experiments where the initial part of the trajectory consists of observed interception positions, through derived best multi-camera setup, and the final part consists of actual positions. The encoder–decoder Bi-LSTM deep neural network is proposed and trained on this dataset. The trained model outperformed the current state-of-the-art by accurately predicting the final 3D catching point, achieving a mean average error of 5 mm and a root-mean-square error of 7 mm in 200 real-world test experiments.

Multi-Vehicle Multi-Camera Tracking With Graph-Based Tracklet Features

Multi-target multi-camera tracking (MTMCT) is an important application in intelligent transportation systems (ITS). The conventional works follow the tracking-by-detection scheme and use the information of the object image separately while matching the object from different cameras. As a result, the association information from the object image is lost. To utilize this information, we propose an efficient MTMCT application that builds features in the form of a graph and customizes graph similarity to match the vehicle objects from different cameras. We present algorithms for both the online scenario, where only the past images are used to match a vehicle object, and the offline scenario, where a given vehicle object is tracked with past and future images. For offline scenarios, our method achieves an IDF1-score of 0.8166 on the Cityflow dataset, which contains the actual scenes of the city from multiple street cameras. For online scenarios, our method achieves an IDF1-score of 0.75 with an FPS of 14. Our codes and datasets are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/elituan/GraphBasedTracklet_MTMCT</uri> .

Robust Oral Localization Based on Multicamera Tracking of Self-Identifying Markers

Robust Oral Localization Based on Multicamera Tracking of Self-Identifying Markers

Camera-aware re-identification feature for multi-target multi-camera tracking

Multi-target multi-camera tracking is an important research topic in intelligent surveillance to achieve re-identification (Re-ID) of moving targets across different cameras. However, Re-ID faces significant challenges owing to variations in cameras/viewpoints, making it difficult to learn discriminative feature representations for targets captured from different cameras/viewpoints. A new pipeline is introduced by balancing the across-cameras sample feature space in camera-aware Re-ID framework. Specifically, different sampling strategies play a crucial role on the performance under the same baseline and identify the feature discrepancy between cameras/viewpoints as a crucial factor. This proposed sampling strategy is called camera equalization sampling to learn enhanced feature disparities, which balances cameras under identity rather than randomly sampling in a batch. The sampled images are combined with non visual cues(cameras position encoding) to reduce intra class variance. For the mechanism of camera equalization sampling, the improved camera centric loss function can better reduce the negative impact of individual samples and provide stable learnable features.Our proposed method consists of three modules. (i) Camera equalization (CE) ensures that each batch collects at least one image from each camera for every identity, thereby enabling robust features. (ii) Camera position embedding (CPE) mitigates the scene bias caused by different cameras/viewpoints by encoding camera indices. (iii) Camera Center triplet loss(CCL) based on CE improves higher robustness to outliers and noisy labels. We demonstrated the proposed method's effectiveness on popular datasets such as DukeMTMC-reID, Market-1501, and MSMT17, achieving state-of-the-art performance.

Fast online multi-target multi-camera tracking for vehicles

Fast online multi-target multi-camera tracking for vehicles

Multicamera edge-computing system for persons indoor location and tracking

This paper presents an indoor person localization and tracking system that uses multiple smart cameras equipped with artificial intelligence (AI) accelerators serving as edge-computing nodes. Our main contributions are as follows: (a) the development of a new multicamera tracking system for indoor scenarios; (b) the release of a multitarget multicamera tracking dataset; and (c) the development of an annotation mechanism based on waypoints. The system can simultaneously track several individuals while preserving their privacy and anonymity, because no images or sensitive data are transmitted outside the edge nodes. Only the position and appearance of each person were transmitted to the central server. In addition, a multitarget multicamera tracking dataset was released. The dataset contains recordings from five cameras in an indoor scenario and is annotated with the real-world coordinates of individuals. Ground-truth annotations were semiautomatically generated using a mechanism in which people equipped with mobile phones followed specific paths with predefined waypoints. Software related to the ground-truth annotation mechanism has also been released as open source.

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

Multi-camera multi-object tracking: A review of current trends and future advances

The nascent applicability of multi-camera tracking (MCT) in numerous real-world applications makes it a significant computer vision problem. While visual tracking of objects, especially in video obtained from single camera setup, has drawn huge research attention, the constant identification and tracking of targets as they transit across multiple cameras remains an open research problem. In addition to the linking of target appearance and trajectory information across frames, effective association of such data across multiple cameras is also very critical in MCT. Occlusion, appearance variability, camera motion, as well as nonrigid object structure and motion are widely recognized constraints and major sources of concerns in MCT. In recent years, several literatures have been contributed suggesting a variety of approaches to addressing various problems in MCT. However, studies that critically review and report the advances and trends of research in MCT are still limited. This current study presents a comprehensive and up-to-date review of visual object tracking in multi-camera settings. In this paper, we analyze and categorize existing works based on six crucial facets: problem formulation, adopted problem solving approach, data association requirements, mutual exclusion constraints, benchmark datasets, and performance metrics. Furthermore, the study summarizes the outcomes of 30 state-of-the-art MCT algorithms on common datasets to allow quantitative comparison and analysis of their experimental results. Finally, we examine recent advances in MCT and suggest some promising future research directions.

ALSA: Adaptive Low-light Correction and Self-Attention Module for Vehicle Re-Identification

Multi-Camera Vehicle Re-identification and Tracking (MCVRT) is a challenging task that involves identifying and tracking vehicles across multiple camera views in a surveillance network. Multi-Target Multi-Camera Tracking (MTMCT) and vehicle Re-Identification (Re-ID) are two major technologies applied to MCVRT tasks. Variations in aspect ratio, occlusion, orientation, and lighting conditions make vehicle re-identification and multi-camera tracking challenging. While some existing methods address these problems, it remains a significant challenge in the field. Additionally, most Re-ID datasets only include images captured in well-lit environments, and the impact of dark images on the performance of existing models needs to be explored further. This paper presents a new approach to address the challenge of low-light images in vehicle re-identification and achieves state-of-the-art results on public datasets. Our approach is based on two key components: (i) an Adaptive Low-light correction and Self-Attention module (ALSA) for image pre-processing in Vehicle Re-ID networks, and (ii) a new loss function called Log Triplet Loss (LT-Loss). We evaluated the presented approach through computer simulations on the VeRi-776 dataset, and the results showed that our model achieved a Rank@1 accuracy of 98.99%, and also outperformed commonly used models on dark images. Our study highlights the importance of considering lighting conditions in vehicle re-identification and provides a new approach to address this challenge.

Indoor Multipedestrian Multicamera Tracking Based on Fine Spatiotemporal Constraints

As indoor space is the primary place for pedestrian activities, obtaining intelligent monitoring of indoor pedestrians is crucial for intelligent video surveillance. Previous studies have verified the effectiveness of spatio-temporal constraints in multitarget multi-camera tracking (MTMCT). Pedestrians are generally subjected to fine spatio-temporal constraints within buildings, based on which the indoor geographic information system (GIS) technology can obtain automatic spatio-temporal modeling. Combined with artificial intelligence (AI) technology, we established a research framework of "GIS+AI+IMPMCT." Specifically, we proposed indoor multi-pedestrian multi-camera tracking (IMPMCT) based on fine spatio-temporal constraints. First, we used GIS to map the indoor monitoring images of buildings and automatically model the fine spatio-temporal relationship among the semantics of the entrance of the surveillance zone. Subsequently, we used the machine learning model of pedestrian localization and tracking to obtain local trajectories of pedestrians and combined them with map information to extract entrance semantics of trajectories. Finally, we used the local trajectory semantics and surveillance entrance semantic constraints to obtain a fine spatio-temporal constraint weight matrix between trajectories and fused pedestrian’s apparent features to obtain trajectory matching results. To verify our method, we established an IMPMCT dataset containing fine indoor spatio-temporal information. Our method obtained an IDF1 of 0.805, which is better than those of other methods. Furthermore, the tracking results obtained by the proposed method contained both image space and geospatial trajectories.

Real-Time Multi-Vehicle Multi-Camera Tracking with Graph-Based Tracklet Features

An essential application in intelligent transportation systems is multi-target multi-camera tracking (MTMCT), where the target’s activity is tracked from different cameras. Although the tracking-by-detection scheme is the primary paradigm in MTMCT, the object association information from the video frames is lost. This is mainly because the multi-camera multi-object matching uses the information from the video frames separately. To solve this problem and leverage this association information, we propose an MTMCT framework, where features are built in the form of a graph and a graph similarity algorithm is used to match multi-camera objects. In this paper, we focus on the real-time scenario, where only the past images are used to match an object. Our method achieves an IDF1 score (the ratio of the number of correctly identified objects to the number of ground truth and average objects) of 0.75 with a rate of 14 frames per second (fps).

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

Multi-Object Multi-Camera Tracking Based on Deep Learning for Intelligent Transportation: A Review.

Multi-Objective Multi-Camera Tracking (MOMCT) is aimed at locating and identifying multiple objects from video captured by multiple cameras. With the advancement of technology in recent years, it has received a lot of attention from researchers in applications such as intelligent transportation, public safety and self-driving driving technology. As a result, a large number of excellent research results have emerged in the field of MOMCT. To facilitate the rapid development of intelligent transportation, researchers need to keep abreast of the latest research and current challenges in related field. Therefore, this paper provide a comprehensive review of multi-object multi-camera tracking based on deep learning for intelligent transportation. Specifically, we first introduce the main object detectors for MOMCT in detail. Secondly, we give an in-depth analysis of deep learning based MOMCT and evaluate advanced methods through visualisation. Thirdly, we summarize the popular benchmark data sets and metrics to provide quantitative and comprehensive comparisons. Finally, we point out the challenges faced by MOMCT in intelligent transportation and present practical suggestions for the future direction.

Effect of Increasing the Number of Substitutions on Physical Performance during Periods of Congested Fixtures in Football

(I) This study aimed to evaluate the impact on physical demands induced by FIFA's new rule implemented based on the number of substitutions caused by COVID-19. (II) Sixty-six matches were analysed in peak periods (microcycles of three matches in a week) in the competition period before and after the pandemic. The variables collected were organised by team (22 from LaLigaTM SmartBank 2019-2020) for a total of 132 team records and 1077 player performance reports using a multi-camera tracking system and Mediacoach® software. Physical performance variables were analysed in the first half, second half and whole match, thus determining the individual and collective performances of the team. (III) This study shows how, despite the increase in substitutions allowed with the new rule, physical performance increased in some variables in the congested periods (e.g., total distance run and distance run in the first and second halves). Additionally, the players' physical performance involved in a substitution was greater than it was for players who completed the game. (IV) The new substitution rule helps to maintain and even improve physical performance. This measure could improve intensity levels in both individual and team performance. It could even safeguard the physical integrity of the players by reducing the risk of injury, as fewer players have to play the full match.