Object Tracking Research Articles

Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration.

Tracking body parts in behaving animals, extracting fluorescence signals from cells embedded in deforming tissue, and analyzing cell migration patterns during development all require tracking objects with partially correlated motion. As dataset sizes increase, manual tracking of objects becomes prohibitively inefficient and slow, necessitating automated and semi-automated computational tools. Unfortunately, existing methods for multiple object tracking (MOT) are either developed for specific datasets and hence do not generalize well to other datasets, or require large amounts of training data that are not readily available. This is further exacerbated when tracking fluorescent sources in moving and deforming tissues, where the lack of unique features and sparsely populated images create a challenging environment, especially for modern deep learning techniques. By leveraging technology recently developed for spatial transformer networks, we propose ZephIR, an image registration framework for semi-supervised MOT in 2D and 3D videos. ZephIR can generalize to a wide range of biological systems by incorporating adjustable parameters that encode spatial (sparsity, texture, rigidity) and temporal priors of a given data class. We demonstrate the accuracy and versatility of our approach in a variety of applications, including tracking the body parts of a behaving mouse and neurons in the brain of a freely moving C. elegans. We provide an open-source package along with a web-based graphical user interface that allows users to provide small numbers of annotations to interactively improve tracking results.

Anti-Occlusion Visual Tracking Algorithm for UAVs with Multi-Feature Adaptive Fusion

Most of the existing trackers based on discriminative correlation filters use only one feature or a simple linear fusion of multiple features for object tracking, and most of them lack a mechanism to handle occlusions. This leads to poor tracking performance in rapidly changing and easily occluded scenarios, especially on unmanned aerial vehicle (UAV) platforms. To address this issue, this paper proposes an anti-occlusion visual tracking algorithm for UAVs with multi-feature adaptive fusion named multi-feature adaptive fusion and anti-occlusion tracker (MAFAOT). It introduces a novel approach for implementing an adaptive fusion of multiple features. This method transforms the multi-feature fusion problem into a maximization issue by designing a tracking quality evaluation index. It successfully achieves an adaptive fusion of gradient direction histogram and color histogram feature responses. MAFAOT also introduces an anti-occlusion update pool strategy, enabling the tracker to adapt dynamically to various complex scenarios, including occlusion and motion blur. The experimental results on the OTB100 and UAV123 datasets confirm the significant advantages of MAFAOT in terms of precision and success rate compared to other correlation filter-based algorithms. The proposed methods further enhance the expressiveness of the features and effectively avoid the problem of tracker contamination caused by occlusion. Furthermore, this paper applies the proposed methods to the kernelized correlation filters (KCF) algorithm. On the OTB100 dataset, the improved KCF algorithm shows an improvement of 10.94% in precision and 11.11% in success rate. On the UAV123 dataset, it shows an improvement of 14.53% in precision and 16.62% in success rate, further verifying the effectiveness and versatility of the proposed methods.

BmmW: A DNN-based joint BLE and mmWave radar system for accurate 3D localization with goal-oriented communication

Bluetooth Low Energy (BLE) has emerged as one of the reference technologies for the development of indoor localization systems, due to its increasing ubiquity, low-cost hardware, and to the introduction of direction-finding enhancements improving its ranging performance. However, the intrinsic narrowband nature of BLE makes this technology susceptible to multipath and channel interference. As a result, it is still challenging to achieve decimetre-level localization accuracy, which is necessary when developing location-based services for smart factories and workspaces. To address this challenge, we present BmmW, an indoor localization system that augments the ranging estimates obtained with BLE5.1’s constant tone extension feature with mmWave radar measurements to provide 3D localization of a mobile tag with decimetre-level accuracy. Specifically, BmmW embeds a deep neural network (DNN) that is jointly trained with both BLE and mmWave measurements, practically leveraging the strengths of both technologies. In fact, mmWave radars can locate objects and people with decimetre-level accuracy, but their effectiveness in monitoring stationary targets and multiple objects is limited, and they also suffer from a fast signal attenuation limiting the usable range to a few meters. We evaluate BmmW’s performance experimentally, and show that its joint DNN training scheme allows to track mobile tags with a mean 3D localization accuracy of 10 cm when combining angle-of-arrival BLE measurements with mmWave radar data. We further assess two variations of BmmW: BmmW-Lite and BmmW-Lite+, both tailored for single-antenna BLE devices, which eliminates the necessity for bulky and expensive multi-antenna arrays and represents a cost-effective solution that is easy to integrate into compact IoT devices. In contrast to classic BmmW (which utilizes angle-of-arrival info), BmmW-Lite uses raw in-phase/quadrature (I/Q) measurements, and achieves a mean localization accuracy of 36 cm, thus facilitating precise object tracking in indoor environments even when using budget-friendly single-antenna BLE devices. BmmW-Lite+ extends BmmW-Lite by allowing the localization task to be transferred from the edge to the cloud due to device memory and power constraints. To this end, BmmW-Lite+ employs a goal-oriented communication paradigm that compresses initial BLE features into a more compact semantic format at the edge device, which allows to minimize the amount of data that needs to be sent to the cloud. Our experimental results show that BmmW-Lite+ can compress raw BLE features by up to 12% of their initial size (hence allowing to save network bandwidth and minimize energy consumption), with negligible impact on the localization accuracy.

Synthetic Data for Video Surveillance Applications of Computer Vision: A Review

In recent years, there has been a growing interest in synthetic data for several computer vision applications, such as automotive, detection and tracking, surveillance, medical image analysis and robotics. Early use of synthetic data was aimed at performing controlled experiments under the analysis by synthesis approach. Currently, synthetic data are mainly used for training computer vision models, especially deep learning ones, to address well-known issues of real data, such as manual annotation effort, data imbalance and bias, and privacy-related restrictions. In this work, we survey the use of synthetic training data focusing on applications related to video surveillance, whose relevance has rapidly increased in the past few years due to their connection to security: crowd counting, object and pedestrian detection and tracking, behaviour analysis, person re-identification and face recognition. Synthetic training data are even more interesting in this kind of application, to address further, specific issues arising, e.g., from typically unconstrained image or video acquisition conditions and cross-scene application scenarios. We categorise and discuss the existing methods for creating synthetic data, analyse the synthetic data sets proposed in the literature for each of the considered applications, and provide an overview of their effectiveness as training data. We finally discuss whether and to what extent the existing synthetic data sets mitigate the issues of real data, highlight existing open issues, and suggest future research directions in this field.

Delivery Garbage Behavior Detection Based on Deep Learning

In the context of rapid urbanization, the challenge of effective garbage disposal has become increasingly significant. Traditional methods for addressing illegal littering by pedestrians are not only inefficient but also resource-intensive, demanding considerable manpower and materials. This study introduces a deep learning-based approach for detecting improper garbage disposal behavior. Leveraging advanced deep learning technologies, this approach focuses on object detection, tracking, and human posture analysis to identify and alert against illegal dumping activities captured in video footage, specifically targeting incidents outside designated times or areas. The primary aim is to facilitate prompt detection and mitigate associated health risks. The system uses three deep learning models, YOLOv5, DeepSORT and MobilePose. YOLOv5 is used to identify the human body and garbage bag, DeepSORT is used to track the two, and MobilePose identifies the key points of the human body for posture estimation. The tracking algorithm is used to determine whether to throw garbage.

Optimizing network lifetime: ERBS-REE for resilient object detection and tracking in resource-constrained WSN environments

Optimizing network lifetime: ERBS-REE for resilient object detection and tracking in resource-constrained WSN environments

Optimizing intrinsic representation for tracking

Learning target representations dedicated to tracking tasks is a promising direction in visual object tracking. Current state-of-the-art approaches exploit pixel appearance reconstruction of auto-encoders for representation training. Such pixel ways are redundant for the high-level semantics of tracking and suffer from an optimization gap where better appearance reconstruction does not correspond to better tracking performance. This article explores the problem of reconstruction misalignment of target representations in Siamese network architectures. We propose an optimization framework for intrinsic representation that restores the high-level feature loss between the networks on the reconstruction side and the target side. This intrinsic representation is generated by feeding the masked image tokens to the target side branch. Unlike the pixel ways, our approach learns to reconstruct the semantic latent level of representation for tracking objects. Moreover, we investigate feeding strategies from the information bottleneck perspective for masked patches to increase the masking performance yet reduce the computational complexity. Additionally, robust propagation mechanisms for the weight parameters of two-sided networks are also explored. Comprehensive evaluations on seven tracking benchmarks demonstrate our effectiveness: GOT-10k, LaSOT, TrackingNet, NFS, UAV123, TNL2K, and OTB100. Compared to the previous best pixel tracker, we outperform it on all seven datasets, with a maximum absolute score of 5.9% on average overlap and a Graphics Processing Unit (GPU) speed of 109 frames per second.

Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object Tracking

In this work, we are dedicated to multi-target active object tracking (AOT), where the goal is to achieve continuous tracking of targets through real-time control of camera. This form of active camera control can be applied to unmanned aerial vehicles (UAV), intelligent robots, and sports events. Our work is conducted in an environment featuring multiple cameras and targets, where our goal is to maximize target coverage. Contrasting with previous research, our work introduces additional degrees of freedom for the cameras, allowing them not only to rotate but also to move along boundary lines. In addition, we model the motion of target to predict the future position of the target in environment. With target’s future position, we use Monte Carlo Tree Search (MCTS) method to find the optimal action of camera. Since the action space is large, we propose to leverage the action selection from multi-agent reinforcement learning (MARL) network to prune the search tree of Monte Carlo Tree Search method, so as to find the optimal action more efficiently. We establish a multi-target 2D environment to simulate several sports games, and experimental results demonstrate that our method can effectively improve the target coverage. The code is available at: http://github.com/HopeChanger/ActiveObjectTracking .

African Easterly Wave Strength and Observed Atlantic Tropical Cyclone Genesis and Characteristics

AbstractAfrican easterly waves (AEWs) are known precursors to Atlantic tropical cyclones (TCs), and are therefore often directly connected to extreme weather events that can be both deadly and destructive. It is well established that not all AEWs develop into TCs, and there has been substantial research that has addressed the different characteristics and environments of developing and non‐developing waves. In this study, however, we specifically examine 41‐years of developing AEWs to provide a better understanding of the relationship between the developing wave and the environment, and the resulting TC. To conduct this research, we identified TCs with AEW origins from the observational record between 1980 and 2020. We then used an objective tracking algorithm to identify the developing AEWs in reanalysis data. We found a statistically significant relationship between the strength of the developing AEWs, TC genesis location and landfall, and sea surface temperature (SST) during TC genesis. Weaker AEWs tend to develop into TCs closer to the Americas in a region with warmer SSTs than those of the stronger AEWs, which tend to develop into TCs closer to Africa. Consequently, the TCs that develop from weaker AEWs are more likely to make landfall due to the close proximity of their genesis locations to the Americas.

Object Detection for Unmanned Aerial Vehicles: A Comprehensive Review

Goal: Researchers studying artificial intelligence have focused a lot of emphasis on computer vision in drones. Drones with intelligence can tackle a lot of issues in real time. For the purpose of monitoring particular surroundings, computer vision tasks like object identification, object tracking, and object counting are important. It becomes increasingly difficult to do, though, due to elements like motion blur, occlusion, camera angle, and altitude. Methodology: A thorough assessment of the literature on object identification and tracking with unmanned aerial vehicles (UAVs) in relation to various applications has been done for this research. This study highlights the research gaps and provides a summary of the results of previous studies. Contribution: Detailed and categorized object identification techniques are used in UAV photos. A selection of UAV datasets tailored to object identification tasks is provided. Summaries of current research projects in various applications are provided. In order to alleviate highlighted research limitations, a secure onboard processing system on a strong object detection framework in precision agriculture is finally presented.

Perturbation-based dual task assessment in older adults with mild cognitive impairment.

Dual tasking (i.e., concurrent performance of motor and cognitive task) is significantly impaired in older adults with mild cognitive impairment (OAwMCI) compared to cognitively intact older adults (CIOA) and has been associated with increased fall risk. Dual task studies have primarily examined volitionally driven events, and the effects of mild cognitive impairment on reactive balance control (i.e., the ability to recover from unexpected balance threats) are unexplored. We examined the effect of cognitive tasks on reactive balance control in OAwMCI compared to CIOA. Adults >55 years were included and completed the Montreal Cognitive Assessment (MoCA) to categorize them as OAwMCI (MoCA: 18-24, n = 15) or CIOA (MoCA: ≥25, n = 15). Both OAwMCI [MoCA: 22.4 (2.2), 65.4 (6.1) years, 3 females] and CIOA [MoCA: 28.4 (1.3), 68.2 (5.5) years, 10 females] responded to large magnitude stance slip-like perturbations alone (single task) and while performing perceptual cognitive tasks targeting the visuomotor domain (target and tracking game). In these tasks, participants rotated their head horizontally to control a motion mouse and catch a falling target (target game) or track a moving object (track). Margin of stability (MOS) and fall outcome (harness load cell >30% body weight) were used to quantify reactive balance control. Cognitive performance was determined using performance error (target) and sum of errors (tracking). A 3 × 2 repeated measures ANOVA examined the effect of group and task on MOS, and generalized estimating equations (GEE) model was used to determine changes in fall outcome between groups and tasks. 2 × 2 repeated measures ANOVAs examined the effect of group and task on cognitive performance. Compared to CIOA, OAwMCI exhibited significantly deteriorated MOS and greater number of falls during both single task and dual task (p < 0.05), and lower dual task tracking performance (p < 0.01). Compared to single task, both OAwMCI and CIOA exhibited significantly deteriorated perceptual cognitive performance during dual task (p < 0.05); however, no change in MOS or fall outcome between single task and dual task was observed. Cognitive impairment may diminish the ability to compensate and provide attentional resources demanded by sensory systems to integrate perturbation specific information, resulting in deteriorated ability to recover balance control among OAwMCI.

Stereoscopic artificial compound eyes for spatiotemporal perception in three-dimensional space.

Arthropods' eyes are effective biological vision systems for object tracking and wide field of view because of their structural uniqueness; however, unlike mammalian eyes, they can hardly acquire the depth information of a static object because of their monocular cues. Therefore, most arthropods rely on motion parallax to track the object in three-dimensional (3D) space. Uniquely, the praying mantis (Mantodea) uses both compound structured eyes and a form of stereopsis and is capable of achieving object recognition in 3D space. Here, by mimicking the vision system of the praying mantis using stereoscopically coupled artificial compound eyes, we demonstrated spatiotemporal object sensing and tracking in 3D space with a wide field of view. Furthermore, to achieve a fast response with minimal latency, data storage/transportation, and power consumption, we processed the visual information at the edge of the system using a synaptic device and a federated split learning algorithm. The designed and fabricated stereoscopic artificial compound eye provides energy-efficient and accurate spatiotemporal object sensing and optical flow tracking. It exhibits a root mean square error of 0.3 centimeter, consuming only approximately 4 millijoules for sensing and tracking. These results are more than 400 times lower than conventional complementary metal-oxide semiconductor-based imaging systems. Our biomimetic imager shows the potential of integrating nature's unique design using hardware and software codesigned technology toward capabilities of edge computing and sensing.

Air Handwriting using AI and ML

Air-writing refers to virtually writing linguistic characters through hand gestures in three dimensional space with six degrees of freedom. In this paper a generic video camera dependent convolutional neural network (CNN) based air-writing framework has been proposed. Gestures are performed using a marker of fixed color in front of a generic video camera followed by color based segmentation to identify the marker and track the trajectory of marker tip. A pre-trained CNN is then used to classify the gesture. The recognition accuracy is further improved using transfer learning with the newly acquired data. The performance of the system varies greatly on the illumi nation condition due to color based segmentation. In a less fluctuating illumination condition the system is able to recognize isolated unistroke numerals of multiple languages. The proposed framework achieved 97.7recognition rate in person inde pendent evaluation over English, Bengali and Devanagari numerals, respectively. Object tracking is considered as an important task within the field of Computer Vision. The invention of faster computers, availability of inexpensive and good quality video cameras and demands of automated video analysis has given popularity to object tracking techniques. Generally, video analysis procedure has three major steps: firstly, detecting of the object, secondly tracking its movement from frame to frame and lastly analysing the behaviour of that object. For object tracking, four different issues are taken into account; selection of suitable object representation, feature selection for tracking, object detection and object tracking. In real world, Object tracking algorithms are the primarily part of different applications such as: automatic surveillance, video indexing and vehicle navigation etc. The generated text can also be used for various purposes, such as sending messages, emails, etc. It will be a powerful means of communication for the deaf. It is an effective communication method that reduces mobile and laptop usage by eliminating the need to write. Key Words: Air Writing, Character Recognition, Object Detection, Real-Time Gesture Control System, Computer Vision , Hand tracking.

Behavior analysis of juvenile steelhead trout under blue and red light color conditions based on multiple object tracking

Behavior analysis of juvenile steelhead trout under blue and red light color conditions based on multiple object tracking

High-precision tracking and positioning for monitoring Holstein cattle.

Enhanced animal welfare has emerged as a pivotal element in contemporary precision animal husbandry, with bovine monitoring constituting a significant facet of precision agriculture. The evolution of intelligent agriculture in recent years has significantly facilitated the integration of drone flight monitoring tools and innovative systems, leveraging deep learning to interpret bovine behavior. Smart drones, outfitted with monitoring systems, have evolved into viable solutions for wildlife protection and monitoring as well as animal husbandry. Nevertheless, challenges arise under actual and multifaceted ranch conditions, where scale alterations, unpredictable movements, and occlusions invariably influence the accurate tracking of unmanned aerial vehicles (UAVs). To address these challenges, this manuscript proposes a tracking algorithm based on deep learning, adhering to the Joint Detection Tracking (JDT) paradigm established by the CenterTrack algorithm. This algorithm is designed to satisfy the requirements of multi-objective tracking in intricate practical scenarios. In comparison with several preeminent tracking algorithms, the proposed Multi-Object Tracking (MOT) algorithm demonstrates superior performance in Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), and IDF1. Additionally, it exhibits enhanced efficiency in managing Identity Switches (ID), False Positives (FP), and False Negatives (FN). This algorithm proficiently mitigates the inherent challenges of MOT in complex, livestock-dense scenarios.

A modular motion compensation pipeline for prospective respiratory motion correction of multi-nuclear MR spectroscopy

Magnetic resonance (MR) acquisitions of the torso are frequently affected by respiratory motion with detrimental effects on signal quality. The motion of organs inside the body is typically decoupled from surface motion and is best captured using rapid MR imaging (MRI). We propose a pipeline for prospective motion correction of the target organ using MR image navigators providing absolute motion estimates in millimeters. Our method is designed to feature multi-nuclear interleaving for non-proton MR acquisitions and to tolerate local transmit coils with inhomogeneous field and sensitivity distributions. OpenCV object tracking was introduced for rapid estimation of in-plane displacements in 2D MR images. A full three-dimensional translation vector was derived by combining displacements from slices of multiple and arbitrary orientations. The pipeline was implemented on 3 T and 7 T MR scanners and tested in phantoms and volunteers. Fast motion handling was achieved with low-resolution 2D MR image navigators and direct implementation of OpenCV into the MR scanner’s reconstruction pipeline. Motion-phantom measurements demonstrate high tracking precision and accuracy with minor processing latency. The feasibility of the pipeline for reliable in-vivo motion extraction was shown on heart and kidney data. Organ motion was manually assessed by independent operators to quantify tracking performance. Object tracking performed convincingly on 7774 navigator images from phantom scans and different organs in volunteers. In particular the kernelized correlation filter (KCF) achieved similar accuracy (74%) as scored from inter-operator comparison (82%) while processing at a rate of over 100 frames per second. We conclude that fast 2D MR navigator images and computer vision object tracking can be used for accurate and rapid prospective motion correction. This and the modular structure of the pipeline allows for the proposed method to be used in imaging of moving organs and in challenging applications like cardiac magnetic resonance spectroscopy (MRS) or magnetic resonance imaging (MRI) guided radiotherapy.

An Image Analysis of River-Floating Waste Materials by Using Deep Learning Techniques

Plastic pollution in the ocean is a severe environmental problem worldwide because rivers carry plastic waste from human activities, harming the ocean’s health, ecosystems, and people. Therefore, monitoring the amount of plastic waste flowing from rivers and streams worldwide is crucial. In response to this issue of river-floating waste, our present research aimed to develop an automated waste measurement method tailored for real rivers. To achieve this, we considered three scenarios: clear visibility, partially submerged waste, and collective mass. We proposed the use of object detection and tracking techniques based on deep learning architectures, specifically the You Only Look Once (YOLOv5) and Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT). The types of waste classified in this research included cans, cartons, plastic bottles, foams, glasses, papers, and plastics in laboratory flume experiments. Our results demonstrated that the refined YOLOv5, when applied to river-floating waste images, achieved high classification accuracy, with 88% or more for the mean average precision. The floating waste tracking using DeepSORT also attained F1 scores high enough for accurate waste counting. Furthermore, we evaluated the proposed method across the three different scenarios, each achieving an 80% accuracy rate, suggesting its potential applicability in real river environments. These results strongly support the effectiveness of our proposed method, leveraging the two deep learning architectures for detecting and tracking river-floating waste with high accuracy.

TomFi: Small Object Tracking Using Commodity WiFi

Rodent infestation has always been one of the most severe threats to humans, which to solve consumes massive manpower and resources. People usually use traps or poisons to treat rat infestation. Such passive countermeasures are inefficient. Damage often occurs when the rats are finally trapped or killed, not to mention the potential risk to injure humans or cause pollution. In this article, we propose a WiFi-based active small object tracking system named TomFi . The core components of TomFi include several deep learning techniques that bridge rat locations/motions and WiFi signal variations (represented by the channel state information). TomFi first employs a detection model to detect the appearance of the rat and then localize it based on a two-branch localization model. Through the design of the two-branch localization model, to our best knowledge, we are the first to solve the information loss and distortion problem. We conducted extensive experiments in both the laboratory environment and real-world kitchen scenarios. The results show that TomFi can achieve a detection success rate of 99%+ and a centimeter-level localization accuracy in real time.

SCGTracker: object feature embedding enhancement based on graph attention networks for multi-object tracking

Multi-object tracking (MOT) is a task to identify objects in videos, however, objects with similar appearance or occlusion may cause frequent ID switching, which is the main challenge of current MOT. In this paper, we propose a novel self-cross graph neural network-based multi-object tracking method, which we termed as SCGTracker. This method seamlessly integrates object detection and tracking through graph neural networks, building upon the foundation of the JDE paradigm. Specifically, we construct graph structures to capture the correlation between objects in both spatial and temporal dimensions. To further tackle the frequent ID switching problem, we employ an attention mechanism to aggregate object context information within the same frame and across different frames, updating the object information via graph neural networks to derive highly distinctive appearance features. Ultimately, the obtained strongly distinguishable object appearance features serve to mitigate the issue of frequent object ID switches. In experiments conducted on the MOT17 test set, our proposed method yields promising results, achieving a 73% Multiple Object Tracking Accuracy (MOTA) and a 73.2% ID F1 score. Furthermore, it demonstrates a substantial reduction in ID switches compared with state-of-the-art methods.

Segmentation and Tracking Based on Equalized Memory Matching Network and Its Application in Electric Substation Inspection

With the wide application of deep learning, power inspection technology has made great progress. However, substation inspection videos often present challenges such as complex backgrounds, uneven lighting distribution, variations in the appearance of power equipment targets, and occlusions, which increase the difficulty of object segmentation and tracking, thereby adversely affecting the accuracy and reliability of power equipment condition monitoring. In this paper, a pixel-level equalized memory matching network (PEMMN) for power intelligent inspection segmentation and tracking is proposed. Firstly, an equalized memory matching network is designed to collect historical information about the target using a memory bank, in which a pixel-level equalized matching method is used to ensure that the reference frame information can be transferred to the current frame reliably, guiding the segmentation tracker to focus on the most informative region in the current frame. Then, to prevent memory explosion and the accumulation of segmentation template errors, a mask quality evaluation module is introduced to obtain the confidence level of the current segmentation result so as to selectively store the frames with high segmentation quality to ensure the reliability of the memory update. Finally, the synthetic feature map generated by the PEMMN and the mask quality assessment strategy are unified into the segmentation tracking framework to achieve accurate segmentation and robust tracking. Experimental results show that the method performs excellently on real substation inspection scenarios and three generalized datasets and has high practical value.