Object Tracking Benchmark Research Articles

Tracking via context-aware regression correlation filter with a spatial–temporal regularization

Correlation filters are known to have superior performance in tracking speed. To improve tracking results, we propose a context-aware regression correlation filter with a spatial–temporal regularization for tracking. First, the spatial regularization parameters are computed by the spatial correlation between the target and the surrounding information. Meanwhile, the filter weight distribution is also computed to highlight the target region and suppress the background region. Next, a context-aware model is proposed to adaptively expand the search area of the target with the original regression analysis. Due to the introduction of the context-aware information, the target structure will be changed and the regression model will not be adapted to the Gaussian function. Therefore, an optimized regression objective function is constructed according to the context-aware model. In the tracking process, under rotation, out-of-view, deformation, etc., the tracker still can continue the tracking of subsequent frames according to partial background information. In addition, time variables for online updating of the spatial–temporal model employ the similarity of adjacent frames to achieve more accurate tracking. According to the model form, the alternative direction method of multipliers is used to optimize the model optimization. Extensive experimental results on the OTB-2013 and OTB-2015 dataset object tracking benchmarks prove that the proposed algorithm is superior to other state-of-the-art algorithms in terms of success, accuracy, and robustness.

HKSiamFC: Visual-Tracking Framework Using Prior Information Provided by Staple and Kalman Filter.

In the field of visual tracking, trackers based on a convolutional neural network (CNN) have had significant achievements. The fully-convolutional Siamese (SiamFC) tracker is a typical representation of these CNN trackers and has attracted much attention. It models visual tracking as a similarity-learning problem. However, experiments showed that SiamFC was not so robust in some complex environments. This may be because the tracker lacked enough prior information about the target. Inspired by the key idea of a Staple tracker and Kalman filter, we constructed two more models to help compensate for SiamFC’s disadvantages. One model contained the target’s prior color information, and the other the target’s prior trajectory information. With these two models, we design a novel and robust tracking framework on the basis of SiamFC. We call it Histogram–Kalman SiamFC (HKSiamFC). We also evaluated HKSiamFC tracker’s performance on dataset of the online object tracking benchmark (OTB) and Temple Color (TC128), and it showed quite competitive performance when compared with the baseline tracker and several other state-of-the-art trackers.

A light tracker for online multiple pedestrian tracking

We propose a novel real-time multiple pedestrian tracker for videos acquired from both static and moving cameras in unconstrained real-world environment. In such scenes, trackers always suffer from noisy detections and frequent occlusions. Existing methods usually use complex learning approaches and a large number of training samples to get discriminative appearance features. However, this leads to high computational cost and hardly works in occlusions (missing detections) and undistinguishable appearance. Addressing this, we design a light two-stage tracker. Firstly, a shallow net with two layers of full convolution is proposed to encode appearance. Compared with other deep architectures and sophisticated learning approaches, our shallow net is efficient and robust enough without any online updating. Secondly, we design a motion model to deal with noisy detections and missing objects caused by motion blur or occlusion. By mining the motion pattern, our tracker can reliably predict the object location under challenging scenes. Furthermore, we propose a speedup version to verify our robustness and the possibility of using in online applications. Extensive experiments are implemented on multiple object tracking benchmarks, MOT15 and MOT17. The performance is competitive over a number of state-of-the-art trackers and demonstrates that our tracker is very promising for real-time applications.

A quantitative attribute-based benchmark methodology for single-target visual tracking

In the past several years, various visual object tracking benchmarks have been proposed, and some of them have been used widely in numerous recently proposed trackers. However, most of the discussions focus on the overall performance, and cannot describe the strengths and weaknesses of the trackers in detail. Meanwhile, several benchmark measures that are often used in tests lack convincing interpretation. In this paper, 12 frame-wise visual attributes that reflect different aspects of the characteristics of image sequences are collated, and a normalized quantitative formulaic definition has been given to each of them for the first time. Based on these definitions, we propose two novel test methodologies, a correlation-based test and a weight-based test, which can provide a more intuitive and easier demonstration of the trackers’ performance for each aspect. Then these methods have been applied to the raw results from one of the most famous tracking challenges, the Video Object Tracking (VOT) Challenge 2017. From the tests, most trackers did not perform well when the size of the target changed rapidly or intensely, and even the advanced deep learning based trackers did not perfectly solve the problem. The scale of the targets was not considered in the calculation of the center location error; however, in a practical test, the center location error is still sensitive to the targets’ changes in size.

Robust long-term object tracking with adaptive scale and rotation estimation

In this article, a robust long-term object tracking algorithm is proposed. It can tackle the challenges of scale and rotation changes during the long-term object tracking for security robots. Firstly, a robust scale and rotation estimation method is proposed to deal with scale changes and rotation motion of the object. It is based on the Fourier–Mellin transform and the kernelized correlation filter. The object’s scale and rotation can be estimated in the continuous space, and the kernelized correlation filter is used to improve the estimation accuracy and robustness. Then a weighted object searching method based on the histogram and the variance is introduced to handle the problem that trackers may fail in the long-term object tracking (due to semi-occlusion or full occlusion). When the tracked object is lost, the object can be relocated in the whole image using the searching method, so the tracker can be recovered from failures. Moreover, two other kernelized correlation filters are learned to estimate the object’s translation and the confidence of tracking results, respectively. The estimated confidence is more accurate and robust using the dedicatedly designed kernelized correlation filter, which is employed to activate the weighted object searching module, and helps to determine whether the searching windows contain objects. We compare the proposed algorithm with state-of-the-art tracking algorithms on the online object tracking benchmark. The experimental results validate the effectiveness and superiority of our tracking algorithm.

LSTM guided ensemble correlation filter tracking with appearance model pool

Deep learning based visual trackers have the potential to provide good performance for object tracking. Most of them use hierarchical features learned from multiple layers of a deep network. However, issues related to deterministic aggregation of these features from various layers, difficulties in estimating variations in scale or rotation of the object being tracked, as well as challenges in effectively modeling the object’s appearance over long time periods leaves substantial scope to improve performance. In this paper, we propose a tracker that learns correlation filters over features from multiple layers of a VGG network. A correlation filter for an individual layer is used to predict the target location. We adaptively learn the contribution of an ensemble of correlation filters for the final location estimation using an LSTM. An adaptive approach is advantageous as different layers encode diverse feature representations and a uniform contribution would not fully exploit this contrastive information. To this end, we use an LSTM as it encodes the interactions for past appearances which is useful for tracking. Further, the scale and rotation parameters are estimated using respective correlation filters. Additionally, an appearance model pool is used that prevents the correlation filter from drifting. Experimental results achieved on five public datasets — Object Tracking Benchmark (OTB100), Visual Object Tracking (VOT) Benchmark 2016, VOT Benchmark 2017, Tracking Dataset and UAV123 Dataset, reveal that our approach outperforms state of the art approaches for object tracking.

Reliability of response region: A novel mechanism in visual tracking by edge computing for IIoT environments

Reliability has been widely used in industrial IoT (IIoT) applications. Since maintaining fast and accurate tracking of targets with fast move and motion blur in industrial applications is still a major challenge, this paper proposes a novel mechanism based on reliability for target matching, which is the basic problem in computer vision. Then, by using the proposed reliability-based mechanism, a novel visual tracking method with edge computing is proposed to achieve accurate and rapid tracking with high reliability. Experimental results on Object Tracking Benchmark (OTB) dataset showed effectiveness of the proposed mechanism by comparing reliability-based and original algorithm. Results also showed that tracking performance of the proposed method has been increased, especially effected greatly on fast-moving, background clutter and motion blur. Therefore, the proposed method is validated to play an important role in real industrial applications with edge computing, which is more suitable for IIoT environments and automotive industry.

Benchmarking Deep Trackers on Aerial Videos.

In recent years, deep learning-based visual object trackers have achieved state-of-the-art performance on several visual object tracking benchmarks. However, most tracking benchmarks are focused on ground level videos, whereas aerial tracking presents a new set of challenges. In this paper, we compare ten trackers based on deep learning techniques on four aerial datasets. We choose top performing trackers utilizing different approaches, specifically tracking by detection, discriminative correlation filters, Siamese networks and reinforcement learning. In our experiments, we use a subset of OTB2015 dataset with aerial style videos; the UAV123 dataset without synthetic sequences; the UAV20L dataset, which contains 20 long sequences; and DTB70 dataset as our benchmark datasets. We compare the advantages and disadvantages of different trackers in different tracking situations encountered in aerial data. Our findings indicate that the trackers perform significantly worse in aerial datasets compared to standard ground level videos. We attribute this effect to smaller target size, camera motion, significant camera rotation with respect to the target, out of view movement, and clutter in the form of occlusions or similar looking distractors near tracked object.

A Scale-Adaptive Particle Filter Tracking Algorithm Based on Offline Trained Multi-Domain Deep Network

In this paper, a MDSPF method is proposed to learn a robust observation model for representing the targets by training a CNN with a number of video sequences. The CNN architecture is composed of three shared convolutional units, two shared fully connected (Fc) units and a multiple domain Fc unit, and it is offline trained by a multi-domain learning strategy. After training, the shared convolutional units are remained as an observation model for our tracking framework. The features from the shared convolutional units can well adapt to the challenges in tracking tasks. A scale-adaptive particle filter is also proposed in our framework to improve the robustness of particle filter method. Different from most existing particle filter tackers, it can efficiently shepherd each particle towards a more precise location and scale through similarity evaluation. Extensive experiments are conducted on Object Tracking Benchmark (OTB), UAV123 and LaSOT datasets to verify the efficiency of our proposed method.

A Visual Object Tracking Algorithm Based on Improved TLD

Visual object tracking is an important research topic in the field of computer vision. Tracking–learning–detection (TLD) decomposes the tracking problem into three modules—tracking, learning, and detection—which provides effective ideas for solving the tracking problem. In order to improve the tracking performance of the TLD tracker, three improvements are proposed in this paper. The built-in tracking module is replaced with a kernelized correlation filter (KCF) algorithm based on the histogram of oriented gradient (HOG) descriptor in the tracking module. Failure detection is added for the response of KCF to identify whether KCF loses the target. A more specific detection area of the detection module is obtained through the estimated location provided by the tracking module. With the above operations, the scanning area of object detection is reduced, and a full frame search is required in the detection module if objects fails to be tracked in the tracking module. Comparative experiments were conducted on the object tracking benchmark (OTB) and the results showed that the tracking speed and accuracy was improved. Further, the TLD tracker performed better in different challenging scenarios with the proposed method, such as motion blur, occlusion, and environmental changes. Moreover, the improved TLD achieved outstanding tracking performance compared with common tracking algorithms.

A cascaded CNN model for multiple human tracking and re-localization in complex video sequences with large displacement

Human tracking and localization play a crucial role in many applications like accident avoidance, action recognition, safety and security, surveillance and crowd analysis. Inspired by its use and scope, we introduced a novel method for human tracking (one or many) and re-localization in a complex environment with large displacement. The model can handle complex background, variations in illumination, changes in target pose, the presence of similar target and appearance (pose and clothes), the motion of target and camera, occlusion of the target, background variation, and massive displacement of the target. Our model uses three convolutional neural network based deep architecture and cascades their learning such that it improves the overall efficiency of the model. The first network learns the pixel level representation of small regions. The second architecture uses these features and learns the displacement of a region with its category between moved, not-moved, and occluded classes. Whereas, the third network improves the displacement result of the second network by utilizing the previous two learning. We also create a semi-synthetic dataset for training purpose. The model is trained on this dataset first and tested on a subset of CamNeT, VOT2015, LITIV-tracking and Visual Tracker Benchmark database without training with real data. The proposed model yield comparative results with respect to current state-of-the-art methods based on evaluation criteria described in Object Tracking Benchmark, TPAMI 2015, CVPR 2013 and ICCV 2017.

Anti-occlusion object tracking based on correlation filter

Despite remarkable progress, visual object tracking is still a challenging task as objects usually suffer from significant appearance changes, fast motion, and serious occlusion. In this paper, we propose an anti-occlusion correlation filter-based tracking method (AO-CF) for robust visual tracking. We first propose an occlusion criterion based on continuous response values. Based on the criterion, objects are divided into four categories to adaptively identify the occlusion of objects. Then we propose a new detection condition for detecting proposals. When the occlusion criterion is triggered, the re-detection mechanism is executed and the tracker is commanded to stop, and then the re-detector selects the most reliable proposal to reinitialize the tracker. Experimental results show that our method outperforms other state-of-the-art trackers in terms of both precision rate and success rate on the widely used object tracking benchmark dataset. In addition, AO-CF is able to achieve real-time tracking speed.

Moving vehicle tracking based on improved tracking–learning–detection algorithm

This study addresses the tracking–learning–detection (TLD) algorithm for long‐term single‐target tracking of moving vehicle from video streams. The problems leading to tracking failures in existing TLD methods are discovered, and an improved TLD (ITLD) tracking algorithm is proposed which is more robust to object occlusion and illumination variation. A square root cubature Kalman filter (SRCKF) is employed in the tracker of TLD to predict the position of the object when occlusion occurs. Besides, this study introduces fast retina keypoint (FREAK) feature into the tracker to alleviate the instability caused by illumination variation or scale variation. The overlap comparison and the normalised cross‐correlation coefficient (NCC) are introduced to the integrator of the TLD to obtain reliable bounding boxes with improved tracking precision. Experiments are conducted to compare the performance of the state‐of‐the‐art trackers and the proposed method, using the object tracking benchmark that includes 50 video sequences (OTB‐50) and TLD datasets. The experimental results show that the proposed ITLD outperforms on both tracking accuracy and robustness. The proposed method can track a moving vehicle even when it is temporally totally occluded.

An improved CNN framework for detecting and tracking human body in unconstraint environment

Human tracking and localization play a crucial role in many applications like accident avoidance, action recognition, safety and security, surveillance and crowd analysis. Inspired by its use and scope, we introduced a novel method for human tracking (one or many) and re-localization in a complex environment with large displacement. The model can handle complex background, variations in illumination, changes in target pose, the presence of similar target and appearance (pose and clothes), the motion of target and camera, occlusion of the target, background variation, and massive displacement of the target. Our model uses three convolutional neural network based deep architecture and cascades their learning such that it improves the overall efficiency of the model. The first network learns the pixel level representation of small regions. The second architecture uses these features and learns the displacement of a region with its category between moved, not-moved, and occluded classes. Whereas, the third network improves the displacement result of the second network by utilizing the previous two learning. We also create a semi-synthetic dataset for training purpose. The model is trained on this dataset first and tested on a subset of CamNeT, VOT2015, LITIV-tracking and Visual Tracker Benchmark database without training with real data. The proposed model yield comparative results with respect to current state-of-the-art methods based on evaluation criteria described in Object Tracking Benchmark, TPAMI 2015, CVPR 2013 and ICCV 2017.

Independent component analysis with reference multifeature integration for visual object tracking via adaptive update

Tracking methods based on correlation filter have been showing great advantages and huge potential. However, it will lead to tracking drift when the target is disturbed, such as occlusion, illumination variation, deformation, and others. In the pursuit of tracking accuracy and robustness, a feature integration model and an adaptive update method for robust tracking are proposed. First, the fine-tuned convolutional neural networks medium is employed to obtain the deep features. The deep features are integrated with traditional manual features by combining with the independent component analysis with reference method to obtain more discriminative features. Second, in the model update phase, an adaptive updating strategy based on the center shift Euclidean distance of image patch is proposed to reduce the unnecessary calculation in model training so as to speed up the tracking speed. Finally, the tracker proposed is evaluated on online tracking benchmark (OTB-2015) and visual object tracking benchmark (VOT-2016). The experimental results show that the method that integrates deep features with traditional manual features can distinguish background and target better. Thus the proposed VOT algorithm performs better robustness and accuracy against state-of-the-art methods when dealing with challenging environments.

Real-Time Object Tracking with Template Tracking and Foreground Detection Network.

In this paper, we propose a fast and accurate deep network-based object tracking method, which combines feature representation, template tracking and foreground detection into a single framework for robust tracking. The proposed framework consists of a backbone network, which feeds into two parallel networks, TmpNet for template tracking and FgNet for foreground detection. The backbone network is a pre-trained modified VGG network, in which a few parameters need to be fine-tuned for adapting to the tracked object. FgNet is a fully convolutional network to distinguish the foreground from background in a pixel-to-pixel manner. The parameter in TmpNet is the learned channel-wise target template, which initializes in the first frame and performs fast template tracking in the test frames. To enable each component to work closely with each other, we use a multi-task loss to end-to-end train the proposed framework. In online tracking, we combine the score maps from TmpNet and FgNet to find the optimal tracking results. Experimental results on object tracking benchmarks demonstrate that our approach achieves favorable tracking accuracy against the state-of-the-art trackers while running at a real-time speed of 38 fps.

A dual CNN–RNN for multiple people tracking

In this paper, we present a deep learning-based approach, namely a dual CNN–RNN for multiple people tracking. We follow tracking-by-detection paradigm by first training a CNN to measure the similarity of two detection boxes. To solve the data association (DA) problem, we build a graph with nodes as detections and edge costs that are the outputs of a CNN. The general minimum cost lifted multi-cut problem (LMP) and corresponding optimization algorithms are utilized to solve the DA problem. To tackle occlusion and ID-switch problems, an RNN network is proposed to predict the nonlinear motion of people. Moreover, we utilize target motion information to stitch tracklets and build long trajectories. The results of our experiments conducted on Multiple Object Tracking Benchmark 2016 (MOT2016) confirm the efficiency of the proposed algorithm.

MSSTResNet-TLD: A robust tracking method based on tracking-learning-detection framework by using multi-scale spatio-temporal residual network feature model

The performance of tracking task is directly dependent on the appearance features of target object, a robust approach for constructing appearance features is crucial for adaptation the appearance change. To construct an accurate and robust appearance model for visual object tracking, we modify original deep residual learning network architecture and name it Multi-Scale Residual Network (MSResNet). The first video frame image and its related information of the current input video sequence are used to learn a multi-scale appearance model of target object and a loss function is minimized over the appearance features. Meanwhile, spatial information of each video frame and temporal information between successive video frames effectively combine with MSResNet. And thus the features are generated by Multi-Scale Spatio-Temporal Residual Network, which is named MSSTResNet feature model, can adapt to scale variation, illumination variation, background clutters, severe deformation of the target object, and so on. We implement a robust tracking method based on tracking-learning-detection framework by using our proposed MSSTResNet feature model and name it MSSTResNet-TLD tracker. Unlike the previous tracking methods, the MSResNet architecture is not offline pre-trained on a large auxiliary datasets but is directly learned end-to-end with a multi-task loss by using the current input video sequence. Furthermore, the multi-task loss function utilizes the classification loss and regression loss that is more accurate for target localization. Our experimental results demonstrate that the proposed tracking method outperforms the current state-of-the-art tracking methods on Visual Object Tracking Benchmark (VOT-2016), Object Tracking Benchmark (OTB-2015), and Unmanned Aerial Vehicles (UAV20L) test datasets. Furthermore, our MSSTResNet-TLD tracker is faster than previous most trackers based on deep Convolutional Neural Network (ConvNet or CNN) and our tracker is extremely robust to the tiny target object. Our source code is available for download athttps://github.com/binger1225/MSSTResNet-TLD-Tracker.

Learning Adaptive Discriminative Correlation Filters via Temporal Consistency Preserving Spatial Feature Selection for Robust Visual Object Tracking

With efficient appearance learning models, discriminative correlation filter (DCF) has been proven to be very successful in recent video object tracking benchmarks and competitions. However, the existing DCF paradigm suffers from two major issues, i.e., spatial boundary effect and temporal filter degradation. To mitigate these challenges, we propose a new DCF-based tracking method. The key innovations of the proposed method include adaptive spatial feature selection and temporal consistent constraints, with which the new tracker enables joint spatial-temporal filter learning in a lower dimensional discriminative manifold. More specifically, we apply structured spatial sparsity constraints to multi-channel filters. Consequently, the process of learning spatial filters can be approximated by the lasso regularization. To encourage temporal consistency, the filter model is restricted to lie around its historical value and updated locally to preserve the global structure in the manifold. Last, a unified optimization framework is proposed to jointly select temporal consistency preserving spatial features and learn discriminative filters with the augmented Lagrangian method. Qualitative and quantitative evaluations have been conducted on a number of well-known benchmarking datasets such as OTB2013, OTB50, OTB100, Temple-Colour, UAV123, and VOT2018. The experimental results demonstrate the superiority of the proposed method over the state-of-the-art approaches.

Parallel Correlation Filters for Real-Time Visual Tracking.

Correlation filter-based methods have recently performed remarkably well in terms of accuracy and speed in the visual object tracking research field. However, most existing correlation filter-based methods are not robust to significant appearance changes in the target, especially when the target undergoes deformation, illumination variation, and rotation. In this paper, a novel parallel correlation filters (PCF) framework is proposed for real-time visual object tracking. Firstly, the proposed method constructs two parallel correlation filters, one for tracking the appearance changes in the target, and the other for tracking the translation of the target. Secondly, through weighted merging the response maps of these two parallel correlation filters, the proposed method accurately locates the center position of the target. Finally, in the training stage, a new reasonable distribution of the correlation output is proposed to replace the original Gaussian distribution to train more accurate correlation filters, which can prevent the model from drifting to achieve excellent tracking performance. The extensive qualitative and quantitative experiments on the common object tracking benchmarks OTB-2013 and OTB-2015 have demonstrated that the proposed PCF tracker outperforms most of the state-of-the-art trackers and achieves a high real-time tracking performance.