Problem Of Object Tracking Research Articles

Puzzle: taking livestock tracking to the next level

Animal behavior is a critical aspect for a better understanding and management of animal health and welfare. The combination of cameras with artificial intelligence holds significant potential, particularly as it eliminates the need to handle animals and allows for the simultaneous measurement of various traits, including activity, space utilization, and inter-individual distance. The primary challenge in using these techniques is dealing with the individualization of data, known as the multiple object tracking problem in computer science. In this article, we propose an original solution called “Puzzle.” Similar to solving a puzzle, where you start with the border pieces that are easy to position, our approach involves commencing with video sequences where tracking is straightforward. This initial phase aims to train a Convolutional Neural Network (CNN) capable of deriving the appearance clues of each animal. The CNN is then used on the entire video, together with distance-based metrics, in order to associate detections and animal id. We illustrated our method in the context of outdoor goat tracking, achieving a high percentage of good tracking, exceeding 90%. We discussed the impact of different criteria used for animal ID association, considering whether they are based solely on location, appearance, or a combination of both. Our findings indicate that, by adopting the puzzle paradigm and tailoring the appearance CNN to the specific video, relying solely on appearance can yield satisfactory results. Finally, we explored the influence of tracking efficacy on two behavioral studies, estimating space utilization and activity. The results demonstrated that the estimation error remained below 10%. The code is entirely open-source and extensively documented. Additionally, it is linked to a data-paper to facilitate the training of any automatic detection algorithm for goats, with the goal of fostering open access within the deep-learning livestock community.

Lester: Rotoscope Animation through Video Object Segmentation and Tracking

This article introduces Lester, a novel method to automatically synthesize retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks’ contours is simplified with the Douglas–Peucker algorithm. Finally, facial traits, pixelation and a basic rim light effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds. The proposed method provides a more simple and deterministic approach than diffusion models based video-to-video translation pipelines, which suffer from temporal consistency problems and do not cope well with pixelated and schematic outputs. The method is also more feasible than techniques based on 3D human pose estimation, which require custom handcrafted 3D models and are very limited with respect to the type of scenes they can process.

Deep Efficient Data Association for Multi-Object Tracking: Augmented with SSIM-Based Ambiguity Elimination.

Recently, to address the multiple object tracking (MOT) problem, we harnessed the power of deep learning-based methods. The tracking-by-detection approach to multiple object tracking (MOT) involves two primary steps: object detection and data association. In the first step, objects of interest are detected in each frame of a video. The second step establishes the correspondence between these detected objects across different frames to track their trajectories. This paper proposes an efficient and unified data association method that utilizes a deep feature association network (deepFAN) to learn the associations. Additionally, the Structural Similarity Index Metric (SSIM) is employed to address uncertainties in the data association, complementing the deep feature association network. These combined association computations effectively link the current detections with the previous tracks, enhancing the overall tracking performance. To evaluate the efficiency of the proposed MOT framework, we conducted a comprehensive analysis of the popular MOT datasets, such as the MOT challenge and UA-DETRAC. The results showed that our technique performed substantially better than the current state-of-the-art methods in terms of standard MOT metrics.

A Decentralized Consistent Approach for Multi-Robot Simultaneous Localization and Object Tracking

This study focuses on the problem of multi-robot simultaneous localization and object tracking (MR-SLOT), which involves the collaborative effort of multiple mobile robots tracking a moving object on a plain. To address this challenge, this paper introduces a fully decentralized approach based on the extended Kalman filter and covariance intersection (CI), denoted as CI-SLOT. In the proposed approach, each robot can precisely track the robot-to-object correlations. Moreover information exchange occurs solely among one-hop communicating robots, and each robot only maintains the latest estimate of the object and its pose. The simulation results have shown that the performances of CI-SLOT are better than those of the state-of-the-art approaches of comparable processing and communication costs.

Adaptive iterative learning control of soft robot for beating heart tracking

PurposeSoft robots are known for their excellent safe interaction ability and promising in surgical applications for their lower risks of damaging the surrounding organs when operating than their rigid counterparts. To explore the potential of soft robots in cardiac surgery, this paper aims to propose an adaptive iterative learning controller for tracking the irregular motion of the beating heart.Design/methodology/approachIn continuous beating heart surgery, providing a relatively stable operating environment for the operator is crucial. It is highly necessary to use position-tracking technology to keep the target and the surgical manipulator as static as possible. To address the position tracking and control challenges associated with dynamic targets, with a focus on tracking the motion of the heart, control design work has been carried out. Considering the lag error introduced by the material properties of the soft surgical robotic arm and system delays, a controller design incorporating iterative learning control with parameter estimation was used for position control. The stability of the controller was analyzed and proven through the construction of a Lyapunov function, taking into account the unique characteristics of the soft robotic system.FindingsThe tracking performance of both the proportional-derivative (PD) position controller and the adaptive iterative learning controller are conducted on the simulated heart platform. The results of these two methods are compared and analyzed. The designed adaptive iterative learning control algorithm for position control at the end effector of the soft robotic system has demonstrated improved control precision and stability compared with traditional PD controllers. It exhibits effective compensation for periodic lag caused by system delays and material characteristics.Originality/valueTracking the beating heart, which undergoes quasi-periodic and complex motion with varying accelerations, poses a significant challenge even for rigid mechanical arms that can be precisely controlled and makes tracking targets located at the surface of the heart with the soft robot fraught with considerable difficulties. This paper originally proposes an adaptive interactive learning control algorithm to cope with the dynamic object tracking problem. The algorithm has theoretically proved its convergence and experimentally validated its performance at the cable-driven soft robot test bed.

3D object tracking using integral imaging with mutual information and Bayesian optimization

Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging’s depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object’s depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object’s bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object’s depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.

Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension.

Video-based referring expression comprehension is a challenging task that requires locating the referred object in each video frame of a given video. While many existing approaches treat this task as an object-tracking problem, their performance is heavily reliant on the quality of the tracking templates. Furthermore, when there is not enough annotation data to assist in template selection, the tracking may fail. Other approaches are based on object detection, but they often use only one adjacent frame of the key frame for feature learning, which limits their ability to establish the relationship between different frames. In addition, improving the fusion of features from multiple frames and referring expressions to effectively locate the referents remains an open problem. To address these issues, we propose a novel approach called the Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net), which is based on one-stage object detection. Our approach includes a Frame Dense Feature Aggregation module for dense feature learning of adjacent time sequences. Additionally, we propose an Image-Language Cross-Generative Fusion module as the main body of multi-stage learning to generate cross-modal features by calculating the similarity between video and expression, and then refining and fusing the generated features. To further enhance the cross-modal feature generation capability of our model, we introduce a consistency loss that constrains the image-language similarity and language-image similarity matrices during feature generation. We evaluate our proposed approach on three public datasets and demonstrate its effectiveness through comprehensive experimental results.

In Defense of Clip-Based Video Relation Detection.

Video Visual Relation Detection (VidVRD) aims to detect visual relationship triplets in videos using spatial bounding boxes and temporal boundaries. Existing VidVRD methods can be broadly categorized into bottom-up and top-down paradigms, depending on their approach to classifying relations. Bottom-up methods follow a clip-based approach where they classify relations of short clip tubelet pairs and then merge them into long video relations. On the other hand, top-down methods directly classify long video tubelet pairs. While recent video-based methods utilizing video tubelets have shown promising results, we argue that the effective modeling of spatial and temporal context plays a more significant role than the choice between clip tubelets and video tubelets. This motivates us to revisit the clip-based paradigm and explore the key success factors in VidVRD. In this paper, we propose a Hierarchical Context Model (HCM) that enriches the object-based spatial context and relation-based temporal context based on clips. We demonstrate that using clip tubelets can achieve superior performance compared to most video-based methods. Additionally, using clip tubelets offers more flexibility in model designs and helps alleviate the limitations associated with video tubelets, such as the challenging long-term object tracking problem and the loss of temporal information in long-term tubelet feature compression. Extensive experiments conducted on two challenging VidVRD benchmarks validate that our HCM achieves a new state-of-the-art performance, highlighting the effectiveness of incorporating advanced spatial and temporal context modeling within the clip-based paradigm.

Development of a multi-object tracking algorithm with untrained features of object matching

The problem of multiple object tracking is one of the most difficult tasks in computer vision. The article is devoted to a task of multiple object tracking on video footage received from an unmanned aerial vehicle. Unlike a static camera platform, the mobile platform causes an accidental camera movement, which leads to sudden changes in the position, angle and scale of objects. Such aspects considerably hinder efficient object tracking. In this paper, we explore the possibilities of improving the tracking quality in the case of camera movements. We significantly outperform ByteTrack algorithm, one of the best tracking algorithms for the MOT Challenge dataset, on the Visdrone 2019 dataset.

An information fusion method for meta-tracker about online aerospace object tracking

The aerospace target tracking is difficult to achieve due to the dataset is intrinsically rare and expensive, and the complex space background, and the large changes of the target in the size. Meta-learning can better train a model when the data sample is insufficient, and tackle the conventional challenges of deep learning, including the data and the fundamental issue of generalization. Meta-learning can quickly generalize a tracker for new task via a few adapt. In order to solve the strenuous problem of object tracking in aerospace, we proposed an aerospace dataset and an information fusion based meta-learning tacker, and named as IF-Mtracker. Our method mainly focuses on reducing conflicts between tasks and save more task information for a better meta learning initial tracker. Our method was a plug-and-play algorithms, which can employ to other optimization based meta-learning algorithm. We verify IF-Mtracker on the OTB and UAV dataset, which obtain state of the art accuracy than some classical tracking method. Finally, we test our proposed method on the Aerospace tracking dataset, the experiment result is also better than some classical tracking method.

A Learning-Based Object Tracking Strategy Using Visual Sensors and Intelligent Robot Arm

This paper focuses on addressing the visual tracking problem using learning-based methods for object tracking tasks. This problem contains a major difficulty, i.e., how to acquire a satisfactory generalization ability of the developed system? In this paper, firstly, the object state tracking system, including a camera-in-hand, a 3D camera and a Rethink Baxter robot, is introduced. The problem formulation is also presented. Secondly, we propose a Kalman-based estimation strategy to acquire the object’s state. In addition, a learning-based tracking controller is developed using the Gaussian mixture models (GMM) method to steer the robot end-effector to track the mobile object. Thirdly, to guarantee system stability (i.e., the position and velocity errors between the object and end-effector will always converge to zeros), the controller parameter constraints are derived, which is a theoretical contribution of this paper. The controller parameter adjustment is avoided by the proposed training process. Thus, the proposed method becomes easy to implement, which is a practical contribution. Finally, the effectiveness of the proposed method is demonstrated by simulation and experimental examples, and the proposed method has satisfactory generalization ability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper studies object tracking problems for different practical applications, such as industrial cutting, grasping and dynamic monitoring. Different trajectory tracking methods have been widely applied in the industrial area. However, users always complain that when the object or trajectory is changed, the tracking controller more or less needs to be re-adjusted. This re-adjust process always requires professional knowledge and programming experience, and thus a factory must employ some professional engineers. In addition, since the objects may be diverse in shape, color and size, to acquire the accurate object position and velocity, an appropriate solution is necessary. Motivated by the above introductions, this paper aims to develop a learning-based controller to track different complex trajectories without frequent and specific parameter adjustment processes. Firstly, a visual measurement system is developed to quickly find and estimate the position and velocity of an object. Secondly, with the object’s information, the learning from demonstration method (GMM method) is applied for the control policy design. Thirdly, the detailed system stability analysis is presented, and the corresponding controller parameter constraints are derived and considered in the proposed control policy. Subsequently, with the demonstration data, the packaged learning algorithm will automatically compute the controller parameters, and users can change the controller performance only by providing the desired demonstrations. In summary, this paper proposes a systematic object tracking solution, and it may bring a new idea to develop a practical object tracking system, using both the learning-based methods to improve the ability of generalization for tracking different objects.

Exploring fusion strategies for accurate RGBT visual object tracking

We address the problem of multi-modal object tracking in video and explore various options available for fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities, including pixel-level, feature-level and decision-level fusion. Specifically, in contrast to the existing approaches, we propose and develop the paradigm for combining multi-modal information for image fusion at pixel level. At the feature level, two different kinds of fusion strategies are investigated for completeness, i.e., the attention-based online fusion strategy and the offline-trained fusion block. At the decision level, a novel fusion strategy is put forward, inspired by the success of the simple averaging configuration which has shown so much promise. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of the proposed decision fusion method produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). A comprehensive comparison of the innovative pixel and feature-level fusion strategies with the proposed decision-level fusion method highlights the advantages fusing multimodal information at the decision score level. Extensive experimental results on five challenging datasets, i.e., GTOT, VOT-RGBT2019, RGBT234, LasHeR and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. The Code is available at https://github.com/Zhangyong-Tang/DFAT.

Dynamic fry counting based on multi-object tracking and one-stage detection

Fry counting is an essential task in aquaculture that provides valuable information for feedstuff feeding, culture density adjustment, and fish economic benefit estimation. However, most current computer vision-based fry counting methods are designed for statically counting, with a fixed number of fry in a container. As a result, these methods are not suitable for dynamic scenarios, such as flowing water, where the effects of factors like water velocity and uneven lighting can cause errors in counting. To address this issue and improve accuracy, we propose a dynamic fry counting method capable of dealing with large-number fry counting. Specifically, we regard fry counting as a Multiple Object Tracking (MOT) problem based on Tracking-By-Detection (TBD) framework. Firstly, a dataset of streaming fry is constructed for training the detection model, and videos with varying numbers of fry are manually counted for counting tests. Secondly, due to the highly similar appearance of fry, Simple Online and Realtime Tracking (SORT) is combined with You Only Look Once version 5 Nano (YOLOv5-Nano) to perform fry tracking. To achieve stable tracking of fry movement, the SORT algorithm is improved based on multi-matching and trajectory recovery, reaching 82.6 % multi-object tracking accuracy. Finally, fry counting is realized based on the intersection of a counting line and moving trajectories. The results indicate that the proposed counting method achieves 96.4 % accuracy and can be implemented on CPU at 5–10 frames per second (fps) and GPU at 15–35 fps. In conclusion, the proposed method automatically counts the number of fry in videos with higher speed and accuracy, facilitating automated fishery farming.

Template-guided frequency attention and adaptive cross-entropy loss for UAV visual tracking

This paper addresses the problem of visual object tracking for Unmanned Aerial Vehicles (UAVs). Most Siamese trackers are used to regard object tracking as classification and regression problems. However, it is difficult for these trackers to accurately classify in the face of similar objects, background clutters and other common challenges in UAV scenes. So, a reliable classifier is the key to improving UAV tracking performance. In this paper, a simple yet efficient tracker following the basic architecture of the Siamese neural network is proposed, which improves the classification ability from three stages. First, the frequency channel attention module is introduced to enhance the target features via frequency domain learning. Second, a template-guided attention module is designed to promote information exchange between the template branch and the search branch, which can get reliable classification response maps. Third, adaptive cross-entropy loss is proposed to make the tracker focus on hard samples that contribute more to the training process, solving the data imbalance between positive and negative samples. To evaluate the performance of the proposed tracker, comprehensive experiments are conducted on two challenging aerial datasets, including UAV123 and UAVDT. Experimental results demonstrate that the proposed tracker achieves favorable tracking performances in aerial benchmarks beyond 41 frames/s. We conducted experiments in real UAV scenes to further verify the efficiency of our tracker in the real world.

Possibilistic space object tracking under epistemic uncertainty

Bayesian filtering is a popular class of estimation algorithms for addressing the space object tracking problem. Bayesian filters assume a random physical system with known statistics of various uncertainty sources. The major challenge is that the exact knowledge of some random process may not be available for analysis, preventing us from performing a probabilistic characterization of the epistemic uncertainty components. In this paper, we explore the use of the Outer Probability Measures (OPMs) to achieve a faithful uncertainty representation derived from all available yet imperfect information in the process of space object tracking. Leveraging the concepts of OPMs, a refined Possibilistic Admissible Region approach is proposed, in which the initial orbital state is modeled using a novel parameter estimation method. The OPM filter is employed to integrate different types of data sources in the presence of assumed ignorance. The efficacy of the developed method is validated by several space object tracking scenarios using real radar measurements and two-line elements data.

SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networks.

Event cameras are asynchronous and neuromorphically inspired visual sensors, which have shown great potential in object tracking because they can easily detect moving objects. Since event cameras output discrete events, they are inherently suitable to coordinate with Spiking Neural Network (SNN), which has a unique event-driven computation characteristic and energy-efficient computing. In this paper, we tackle the problem of event-based object tracking by a novel architecture with a discriminatively trained SNN, called the Spiking Convolutional Tracking Network (SCTN). Taking a segment of events as input, SCTN not only better exploits implicit associations among events rather than event-wise processing, but also fully utilizes precise temporal information and maintains the sparse representation in segments instead of frames. To make SCTN more suitable for object tracking, we propose a new loss function that introduces an exponential Intersection over Union (IoU) in the voltage domain. To the best of our knowledge, this is the first tracking network directly trained with SNN. Besides, we present a new event-based tracking dataset, dubbed DVSOT21. In contrast to other competing trackers, experimental results on DVSOT21 demonstrate that our method achieves competitive performance with very low energy consumption compared to ANN based trackers with very low energy consumption compared to ANN based trackers. With lower energy consumption, tracking on neuromorphic hardware will reveal its advantage.

Visual Object Detection and Tracking for Internet of Things Devices Based on Spatial Attention Powered Multidomain Network

Internet of Things (IoT) has brought changes in many fields by joining physical space with the cyber space. The IoT devices are becoming increasingly complex. With the rapid deployment of cameras, tasks in IoT like visual information are more important, but IoT devices have limited computing resources, including power, computing ability, storage, etc. Some tasks that might be perfectly normal to perform on a computer would be rather challenging on an IoT device. Therefore, how to maintain acceptable performance while minimizing resources is becoming a more consequential part in IoT. In this article, we aim to solve the problem of object detection and tracking in IoT while minimizing resources. The traditional algorithms need to use a convolutional neural network (CNN) to identify different objects in each frame, and then determine the tracking target from identified objects, which typically requires a lot of computing resources. By incorporating spatial attention, and multidomain network, we proposed a novel algorithm named as spatial attention powered multidomain network (SA-MDNet). By adding the spatial attention mechanism to the original MDNet model, and using multiclass cross-entropy loss, we are able to distinguish the background and the target in different video sequences effectively and efficiently. This novel algorithm achieves similar performance on the OTB 50/100/2013 data sets compared to several state-of-art models, while uses only a fraction of the memory compared to MDNet.

Adaptive mixture approximation for target tracking in clutter

Target tracking is a state estimation problem common in many practical scenarios like air traffic control, autonomous vehicles, marine radar surveillance and so on. In a Bayesian perspective, when phenomena like clutter are present, most existing tracking algorithms must deal with association hypotheses which may grow in number over time. In that case, the posterior state distribution can quickly become computationally intractable, and approximations must necessarily be introduced. In this work, the impact of the number of hypotheses and of reduction procedures is investigated both in terms of computational resources and tracking performances. For this purpose, a recently developed adaptive mixture model reduction algorithm is considered in order to assess its performances when applied to the problem of single object tracking in the presence of clutter and to provide some interesting insights into the addressed problem.

A PID Based Meta-Learning Method About Space Non-Cooperative Active Object Tracking

In recent years, more and more scholars pay attention to the problem of active object tracking, which has a wide range of application scenarios. Active tracking of non-cooperative objects in space was a good application example, which has important implications for the exploration of the space environment. Datasets of the space objects are difficult to collect, thus traditional machine learning is difficult to complete space tasks, which needs train from scratch, sufficient datasets and lots of training time. Thus, we propose a PID based for Meta-Learning method for space non-cooperative active object tracking, which overcome the shortcoming of few space datasets, and save a lot of training time. We incorporate PID into the meta-learning algorithm, past and change information of gradients to update the meta-learning parameters, and reducing greatly the overshoot problem of meta-learning and accelerating the optimization process of meta-learning. Our algorithm can help train a better initialized active object tracking model due to the differentiation and integration of gradients, which can retain more information and improve stability. Our PID based meta-learning method was based on reinforcement learning, which can help achieve automated and optimal object tracking. The initial model of our method can track a wide variety of objects without train from scratch. Through a large number of comparative experiments, our method has good results in tracking speed and tracking accuracy with longer episodes and higher rewards.

Tracking of Elliptical Object with Unknown but Fixed Lengths of Axes

This paper addresses the problem of tracking an elliptical object (e.g., a vehicle or aircraft carrier) with unknown but fixed lengths of axes. In practice, such axis lengths are usually time invariant, but the orientation and kinematics may be time varying. To model this extended object tracking (EOT) problem well, we represent the kinematics and orientation by a random vector, and represent the axis lengths by non-random unknown parameters. We investigate the expectation-maximization (EM) algorithm and propose an EM-based EOT approach, which utilizes the prior information about the invariant lengths and estimates the state and parameters in a unified framework. To reduce computation for real-time applications, we develop a recursive, easy to implement approach. Handy and efficient estimation of axis lengths is developed. Simulation and real-data results are presented to illustrate the effectiveness of our modeling and approach.